feat(data): a relation may be keyed by several columns, so a generator's zone can change by period - #1650
Closed
FBumann wants to merge 2 commits into
Closed
feat(data): a relation may be keyed by several columns, so a generator's zone can change by period#1650FBumann wants to merge 2 commits into
FBumann wants to merge 2 commits into
Conversation
…r's zone can change by period The single-valued map is now one value column per key, whatever the key names: `sum(by=)` groups per condition, `at(by=)` reads back per condition, and a `where:` tests the value at every key dimension. A relation the key does not determine a single column of, and a partitioned shift, sum_back or position grouped by a conditioned map, are still refused at the door. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA
This was referenced Sep 15, 2026
Merging this PR will not alter performance
Comparing Footnotes
|
The language admits a partition through a map keyed on more than the dimension it walks — the walk joins on the rest of the key and the operand carries it. What cannot build it is the group table here, and the sentence now says so and names the walked dimension in its rewrite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA
This was referenced Sep 16, 2026
FBumann
added a commit
that referenced
this pull request
Sep 16, 2026
…#1657) > **Prompt:** Then let's fix the bug first > [!NOTE] > The following content was generated by AI. Closes #1652. A self-map — `rep_of: {columns: {snapshot: snapshot, rep: snapshot}, key: snapshot}`, the language's own example of the form — passed `check` and then raised: polars' `DuplicateError` relationally, linopy's coordinate mismatch eagerly, and on `sum(by=)` the two lanes disagreed, one raising where the other built. The walk named its value column after the dimension it lands on, which for a self-map is the name the key column already holds. The mapping table names its columns **by side** now — `__walk key__`, `__walk value {relation}__` — and the join names its sides (`left_on` / `right_on`), so the landing is aliased to its dimension at the end rather than colliding at the start. On the eager lane the pullback puts the fine labels back as the coordinate: a row lands at the snapshot that *read* a value, not at the one it read. <details><summary>What the probe says, before and after</summary> | | `lps.check` | relational | eager | | --- | --- | --- | --- | | `at(price, by=rep_of, over=rep)` before | passes | `DuplicateError` | `ValueError: Coordinate mismatch` | | `sum(p, by=rep_of, over=snapshot, into=rep)` before | passes | `DuplicateError` | builds, unchecked | | both, after | passes | builds | builds, and the two agree | </details> <details><summary>Mutation table</summary> Baseline for this environment: 10 failures, all `gurobi`/`xpress` not installed. | mutation | result | |---|---| | the one-value-column-per-relation guard (`compiler.py:703-706`) | **caught** | | the walk selects its landing by the dimension's own name (`compiler.py:845`) | **caught** — 218 failed | | the eager pullback keeps the labels it read (`operators.py:120`) | **caught** — 11 failed, 1 beyond the baseline | | the join renames the mapping onto the fragment's dims instead of naming both sides (`compiler.py:846`) | **not caught** — recorded rather than hidden: it is an equivalent implementation, since what the bug needed was the *landing* to keep a name of its own, which the row above is the guard for | The first through `python -m tools.mutate`, which reported the tree clean afterwards; the rest by hand (changed lines, which the tool does not express), each with the same three precautions. </details> <details><summary>Verified</summary> `ruff check` · `ruff format --check` · `pyrefly check` · `pytest -q -n 4` — the suite reaches the same 10 failures as `main` in this environment, all a solver that is not installed. Run in a `uv` venv rather than pixi (no pixi here). **Not run**: `pixi run test-floors` and `test-bench` — no pixi environment here. The bug was reproduced first: `tests/test_self_map.py` landed as two **strict xfails** naming #1652 (commit `0324845`), which XPASS on the fix and lose their markers in the commit that makes them pass. What was wrong is in their docstrings. The `where:` case and the duplicate-key refusal pass beside them, which is what said the data path was never the problem. Coverage that moved: `test_sum_by_relations.test_a_hand_built_walk_onto_two_columns_is_refused` met a `zip` crash inside the mapping builder; the guard it probes is now a named assertion that fires before any relation is read, and the test matches that. </details> ## What I did not do - **The other three shapes stay refused** — #1654, #1655, #1656 under #1653. This is the bug of the four; those are gaps. - **No example or gallery page.** A self-map is exercised by the new test module. - **#1650 touches the same two functions.** It is not a dependency either way — this branches off `main` so it can merge alone — but whichever lands second wants a small merge in `_mapping` and `operator_at`. I will do that when you merge one. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA --- _Generated by [Claude Code](https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
FBumann
added a commit
that referenced
this pull request
Sep 16, 2026
…language admits (#1659) > **Prompt:** "Let's catch up and support everything from mathspecs relations feature. If the linopy lane is to hard, leave it stale. The relational lane is what we care about. Also take the time to potentially refactor the code about relations, as the concept has changed a bit" > [!NOTE] > The following content was generated by AI. The relational lane builds every relation shape the language admits: a key of several columns, several value columns, a bare relation, a partition by a conditioned map, a self-map, and two columns over one dimension. The eager lane keeps the single-valued map and refuses the rest, naming the lane that builds it. Closes #1654, #1655, #1656 and #1653. Supersedes #1650, whose eager-lane half is carried here. #1657 (the self-map fix, #1652) landed on `main` meanwhile and is merged in: the general walk carries its relational half, and the eager lane keeps its `operator_at` fix. <details><summary>What moved</summary> - **The transport.** A relation is attached as the table it declares, one column per column under its own name. `sources.py` checks every column against its dimension, one row per key tuple (every column, for a bare relation), and a null anywhere; `attaching.py` casts every column to its dimension's `Enum`. The archive stores the table as supplied, so `supplied()` no longer renames. - **The compiler** reads the plan's `Walk`. `_walked` puts consumed and joined columns under their dimensions and produced ones under a landing name until the consumed column is dropped, which is what a self-map needs. `_remap_fragment` joins on consumed, joined, and any produced dimension the operand already carries (the masked sum). `_empty_groups` and `_pulled_back_presences` carry the joined dimensions and several produced ones. `partitioned` takes the `Walk` and ranks inside `(joined dims…, group columns…)`. - **Partitions** (`reindex.py`) key every side, edge and presence by the dimension and the partition's joined dimensions; a per-group offset or width is read under the group column. **Predicates** read a relation at every key dimension, a bare existence at every column, and a grouped position at the rest of the key. - **`lpspec/relations.py` is deleted.** Its accessors assumed the two-column map; the language's `Walk` carries the fact, and `lanes.lowered` refuses no relation any more. - **The eager lane** takes #1650's multi-key support (one array per relation over its key's product, `groupby` on value and condition), reads a relation's key columns by role so a self-map's two columns stay apart, and refuses the other three shapes in `linopy/loader.py` with a `LaneError` naming the relational lane. - **The parity tables** (`differential/pypsa/tables`) are regenerated with `parity.py` against the pinned corpus: every rung still matches PyPSA, and the only change is each relation table's header, which `tidy_sources` no longer renames. - **Docs:** `architecture.md` (module rows, the `GroupSum` row, the relational-lane paragraph), `reference/data.md` (relation rows), `about/linopy.md` (a third wall between the lanes). </details> <details><summary>Coverage that moved</summary> - `tests/test_relation_shapes.py` (new): every shape on the relational lane, each optimum hand-derived and the written LP file re-solved to it; the masked sum; a bare relation holding a pair twice. The self-map walks live in `main`'s `test_self_map.py`, which is differential; this module keeps the self-map partition. - `tests/test_conditioned_relations.py` (new): #1650's differential cases for the multi-key map, the eager lane's three refusals, and the door checks. - `test_label_coords.py`: the two refusal cases became "passes check" cases. `test_sum_by_relations.test_a_hand_built_walk_onto_two_columns_is_refused` is deleted, the shape now building in the new module. - Door messages reworded for the general shape, regexes moved with them (`test_self_map.py`'s included): `maps 1 key(s) more than once: generator='g1'`, `has value(s) in 'generator' that are not 'generator' labels`, `null in 'bus'`, `is a relation with a column over 'g'`. </details> <details><summary>Mutation table</summary> Taken with `python -m tools.mutate` on 419870d, before the merge; the tree came back clean. The two-columns-over-one-dimension guard was retired by the merge, the eager lane building the shape since #1657. | mutation | result | |---|---| | the eager lane refuses a bare relation (`loader.py:40-41`) | **caught** | | the eager lane refuses a key determining several columns (`loader.py:42-48`) | **caught** | | the eager lane refuses a partition by a conditioned map (`loader.py:53-60`) | **caught** | | a relation holding a row twice is refused (`sources.py:341-352`) | **caught** | </details> <details><summary>Verified</summary> `ruff check` · `ruff format --check` · `pyrefly check` · `pytest -q -n 4` on the merged head 4d133ea: the suite reaches the same 10 failures as `main` (3e825f0) in this environment, all of them a solver that is not installed (`gurobi`, `xpress`). `differential/pypsa/parity.py` against math-spec at v0.0.0-alpha.89: every rung matches. Run in a `uv` venv with the `linopy` extra rather than pixi (no pixi here). **Not run:** `docs-build` and `test-floors`, no pixi environment here. </details> ## What I did not do - **The eager lane is left at the single-valued map**, as asked. The three other shapes are refused there, not built; their tests use the LP file as the second opinion. - **No new example or gallery page.** A page for a line-ends table is a `docs:` PR of its own. - **Two lanes still each answer "which dimensions do a node's walks join on"** (`compiler.joined_dims`, `builder._joined_dims`). energy-models/mathspec#488 puts `joined` on `GroupSum` and `At`; the two copies go with the pin bump once it releases. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DT1r52zPp9BdnrzFdp1xoY --------- Co-authored-by: Claude <noreply@anthropic.com>
Collaborator
Author
|
Superseeded |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
The following content was generated by AI.
The single-valued map is now one value column per key, whatever the key names.
sum(by=)groups per condition,at(by=)reads back per condition, and awhere:tests the value at every key dimension. What #1567 asked for — a generator's zone that changes by period — is one relation now, not a relation per period.The language already lowered it (math-spec#437's
Walkcarriesjoinedroles);lanes.loweredrefused it. Refused still: a relation with no key, a relation whose key determines two columns, and a partitionedshift/sum_back/positiongrouped by a conditioned map — the last of which the language admits, so its refusal says the limit is this package's group table and names the walked dimension in the rewrite. #1651 carries all three.What moved
relations.py—key_role/key_dimbecomekey_roles/key_dims;refusalturns away only a keyless relation and one with several value columns, and gains the partition rule above, which needs a walk over the plan (_partitioning).sources.py— a relation arrives with one column per key column; the null, duplicate and stray-label checks are per key tuple and per key dimension.compiler.py—_mappingtakes the node's walks and names every column by its dimension;_remap_fragmentjoins on the consumed dims and the joined ones._empty_groupsand the pullback's presences carry the condition too.predicates.py— a relation comparison joins on every key dimension.loader.py), sodim_coordsbecomes a flatrelationsmap;operator_grouped_sumgroups by(value, condition)and masks the operand where a pair is unmapped rather than dropping a label;operator_atfills and masks back for the same reason.Mutation table
Baseline for this environment: 10 failures, all
gurobi/xpressnot installed.relations.py:47-50)compiler.py:700-704)compiler.py:834)operators.py:99)The first two through
python -m tools.mutate, which reported the tree clean afterwards; the last two by hand (a changed line, which the tool does not express), each with the same three precautions — committed tree,git checkout --,__pycache__dropped on both sides.Verified
ruff check·ruff format --check·pyrefly check·pytest -q -n 4— the suite reaches the same 10 failures asmainin this environment, all of them a solver that is not installed (gurobi,xpress). Run in auvvenv rather than pixi (no pixi here).Not run:
pixi run test-floors(the bare-install shape) andtest-bench— no pixi environment here.New coverage in
tests/test_conditioned_relations.py: the grouping, an unmapped pair, the pullback, thewhere, both refusals, and three door checks — the first four differential, the first also against the LP file.Coverage that moved:
test_label_coords.py'sa-key-of-two-columns-joins-on-one-the-walk-does-not-tradewas a refusal case and is now the feature, so it is deleted and its shape is the new module's subject.test_sum_by_relations.test_a_hand_built_walk_onto_two_columns_is_refusednow meets a named assertion rather than azipcrash.What I did not do
_groupedis one-dimensional. Building it is its own change, in a relation with several value columns, and one with no key, are still refused #1651.docs:PR of its own.mainand still does on this branch; the walk's output column needs a name of its own, which is a separate fix.🤖 Generated with Claude Code
https://claude.ai/code/session_014pdMgEVkGBfh1f2AyiCSKA