diff --git a/docs/about/relations-as-linear-maps.md b/docs/about/relations-as-linear-maps.md new file mode 100644 index 00000000..e94118d6 --- /dev/null +++ b/docs/about/relations-as-linear-maps.md @@ -0,0 +1,196 @@ + + +# Relations as linear maps + +This page says what a [relation](../reference/language/relations.md) is in the +language of linear algebra. It then shows that the join and group-by on that +page are one computation with the sum a paper prints. Read it if "joined on" +and "grouped by" read as database words and you want the math they stand for. + +```yaml +dimensions: + snapshot: { dtype: int } + generator: { dtype: str } + zone: { dtype: str } +relations: + gen_zone: { key: [generator, snapshot], values: zone } +parameters: + zone_cap: { dims: [snapshot, zone] } +variables: + p: { dims: [snapshot, generator] } +constraints: + zonal: + dims: [snapshot, zone] + expression: sum(p, by=gen_zone, over=generator, into=zone) <= zone_cap + looked_up: + dims: [snapshot, generator] + where: gen_zone + expression: p <= at(zone_cap, by=gen_zone, over=zone, into=generator) +objective: + sense: minimize + expression: sum(p) +``` + +## A relation is an indicator + +A relation with columns over the dimensions $`D_1, \dots, D_n`$ is a set of +rows, so it is a subset $`R \subseteq D_1 \times \dots \times D_n`$. Its +**indicator** $`\mathbf{1}_R`$ is $`1`$ at a row of the table and $`0`$ +everywhere else. `key:` is a claim about the shape of that set: one row per key +tuple. So a relation with `key: K` and `values: V` is the graph of a function +$`f: K \to V`$, and + +```math +\mathbf{1}_R(k, v) = [\, f(k) = v \,]. +``` + +The function is partial where a key tuple has no row. A bare relation is a +subset and nothing more. Above, `gen_zone` is the graph of +$`f: \mathcal{G} \times \mathcal{T} \to \mathcal{Z}`$. The +[data contract](../reference/language/relations.md#the-data-contract) makes it +one: the loader checks one row per key tuple when the data binds. + +## A join and group-by is a contraction + +`sum(p, by=gen_zone, over=generator, into=zone)` is the product of two arrays, +summed over the one index they share and the call names: + +```math +y_{t,z} = \sum_{g} \mathbf{1}_R(g, t, z) \cdot p_{t,g} = \sum_{g \,:\, f(g,\,t) = z} p_{t,g} +``` + +The right-hand form is what the typesetter +[prints](../reference/notation.md). The left-hand form is a tensor contraction, +and the two questions the relations page asks of a column are the two +positions an index can take in it: + +| The relations page says | In the formula | +| ------------------------ | ----------------------------------------------------------- | +| joined on, `over=` | $`g`$ is on both factors and summed. It leaves. | +| grouped by, `into=` | $`z`$ is on the indicator alone and not summed. It arrives. | +| joined on and grouped by | $`t`$ is on both factors and not summed. It stays. | +| neither | a value column not in the formula. It is not read. | + +**The result carries the free indices.** Those are the indices of the operand +and the indicator together, less the summed one, which is +`(dims(x) − joined) ∪ grouped`. That is the rule in the [expressions +reference](../reference/language/expressions.md#how-dimensions-combine), and +`_join_dims` in `src/math_spec/dimensions.py` computes it. The three refusals +beside it are the three things the formula needs: + +- **The operand carries every column joined on**, or there is nothing to match. +- **The operand carries every unnamed key column.** Otherwise $`t`$ would sit + on the indicator alone, which is the grouped position, and the call names a + grouped column with `into=`. +- **The operand carries no column grouped by.** Otherwise $`z`$ would sit on + both factors and not be summed. That matches the two occurrences instead of + adding an axis. The language makes you write that match outside the + operator: `load * sum(p, by=gen_bus, over=generator, into=bus)`. + +So "joined on" and "grouped by" are the two positions of the summation +convention, applied to one product. Nothing on the relations page is a +separate rule. + +**The unnamed key column makes the matrix block-diagonal.** At each $`t`$, +$`M_t[z, g] = \mathbf{1}_R(g, t, z)`$ is a $`|\mathcal{Z}| \times |\mathcal{G}|`$ +matrix of zeros and ones, and $`y_t = M_t\, p_t`$. A relation keyed by one +column has one block. + +## A join alone is the transpose + +`at(zone_cap, by=gen_zone, over=zone, into=generator)` joins the same table +with the other index bound: + +```math +w_{t,g} = \sum_{z} \mathbf{1}_R(g, t, z) \cdot \mathrm{zone\_cap}_{t,z} = \mathrm{zone\_cap}_{t,\, f(g,\,t)} +``` + +Same indicator, same contraction, so the same free-index rule gives the result's +dimensions. As a matrix it is $`M_t^{\mathsf{T}}`$. Because $`R`$ is the graph +of $`f`$, the sum over $`z`$ has exactly one term where $`(g, t)`$ is in the +domain of $`f`$, and none elsewhere. So the contraction is the composition +$`\mathrm{zone\_cap} \circ f`$, which is a pullback. + +**That one term is the whole difference between `at` and `sum`**, and +resolution decides it from the key alone. Where the columns a call groups by +hold the whole key, every group is one row, and the group-by adds nothing. +`_join` in `src/math_spec/resolution.py` names this `one_row_per_group`. A +`sum` with one row per group adds up nothing and is refused toward `at`. An +`at` with several rows per group would have several terms and is refused toward +`sum`. The +[relations page](../reference/language/relations.md#joins-and-group-bys) +quotes the message. + +**The two are adjoint.** For `gen_bus: { key: generator, values: bus }`, +$`x`$ over generators and $`y`$ over buses, + +```math +\langle M x, y \rangle = \sum_{b} y_b \sum_{g} \mathbf{1}_R(g, b)\, x_g = \sum_{g} x_g \sum_{b} \mathbf{1}_R(g, b)\, y_b = \langle x, M^{\mathsf{T}} y \rangle, +``` + +which is why the program lowers `at` to a `Join` node and `sum(by=)` to the +same `Join` under a `Sum`. The join keeps the column it sums away as an axis +named for the relation's column, `gen_zone.generator`, and the `Sum` stands +over that axis. So a map into its own dimension, which drops and adds one +dimension, still has two axes between the join and the sum. A bare relation has +the same matrix without the functional claim. A column of $`M`$ may hold several +ones, so the sum fans out and no group is one row. That is why `at` through a bare relation is refused. + +## The join and the aggregate + +The language fixes the formula, and an engine decides how to evaluate it. A +table stores $`\mathbf{1}_R`$ as its support: the rows where it is $`1`$. +Multiplying a matrix stored that way by a vector takes two steps. + +1. **Pair each row of the table with each row of the operand that agrees on + every shared index**, here $`g`$ and $`t`$. That is an inner equi-join on + the columns joined on. Each pair is one nonzero product + $`1 \cdot p_{t,g}`$, relabelled by the column grouped by, $`z`$. +2. **Add the pairs that agree on the free indices**, here $`(t, z)`$. That is a + group-by on the result's dimensions with a sum. + +lpspec, the reference engine, runs exactly these two steps. `join_relation` in +`src/lpspec/relational/engines/polars/relations.py` is step 1. It runs one inner +join on the dimensions joined on. A select then drops the dimensions not +grouped by and renames the landing column to the dimension grouped by. Step 2 +is the terminal aggregate in `assembly.py`, a `group_by` over the row's +coordinates with `sum`, run once per constraint after every term has landed. +Until then a sum of linear terms is a list of terms, and adding is +concatenation. `at` runs step 1 against the same table and needs no step 2: +one row per group means no two pairs land on one coordinate. Where several +$`(g, t)`$ share one $`z`$ the join fans out, which is a column of +$`M^{\mathsf{T}}`$ holding several ones. + +So the two pictures are one. The join is the multiplication by an entry of +$`\mathbf{1}_R`$, which is $`1`$ or absent. The group-by is the $`\sum`$. + +## Where the built model departs from the map + +Two positions of the formula have no variable to build, and there the model a +consumer builds is not the matrix. + +- **An empty group is the empty sum.** A zone no generator maps to at $`t`$ has + $`y_{t,z} = 0`$, and the row reads $`0 \le \mathrm{zone\_cap}_{t,z}`$. It + names no variable, so an engine [does not build + it](../reference/language/absence.md#rows-with-no-variable-terms) and reports + the omission. With `>=` the omitted row would have been infeasible. +- **Off the domain there is no value.** $`\mathrm{zone\_cap} \circ f`$ is + undefined where $`f`$ is, so `at` is absent there and [absence + spreads](../reference/language/absence.md#how-absence-travels) to the row. + `where: gen_zone` on `looked_up` writes the domain of $`f`$ on the page, so a + reader sees which rows exist without opening the data. + +## Partitions and tests + +The other two uses of a relation do not contract against $`\mathbf{1}_R`$. + +- **A partition steps inside a fibre.** `shift(x, along=snapshot, offset=1, +by=season_of, within=season)` reads the neighbour $`t'`$ of $`t`$ with + $`f(t') = f(t)`$. The fibres of $`f`$ partition the axis, and the frame does + not change. +- **A test is the indicator itself.** A relation's name in a `where` evaluates + $`\mathbf{1}_R`$ at the frame's own coordinate, and keeps the coordinate where + it is $`1`$. diff --git a/docs/about/what-counts-as-language.md b/docs/about/what-counts-as-language.md index f0f52271..c458c6b2 100644 --- a/docs/about/what-counts-as-language.md +++ b/docs/about/what-counts-as-language.md @@ -29,8 +29,8 @@ Four rules follow from the test: in the renderer. - The set of operators is fixed. A tool cannot add a `roll` that the others do not know. -- Each operator has one rule for the dimensions it produces. `sum(p, by=gen_bus, over=generator, into=bus)` - lands on `bus` for every program. +- Each operator has one rule for the dimensions of its result. `sum(p, by=gen_bus, over=generator, into=bus)` + groups by `bus` for every program. - Degree is decided when the file loads. Whether `x * y` is allowed does not depend on which engine builds the model. diff --git a/docs/contributing.md b/docs/contributing.md index 271bbed5..56f33171 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -107,13 +107,13 @@ suffix says which layer: A node names the operation, not the verb a file writes. One verb can resolve to two nodes, so the file's spelling cannot decide the name. -| File verb | Node | What the node names | -| ------------------ | ----------- | ------------------------------ | -| `sum(over=)` | `Sum` | dims removed from the result | -| `sum(by=)` | `GroupSum` | a sum through a relation | -| `at(by=)` | `Pullback` | a read through a relation | -| `shift(along=)` | `Translate` | a re-index along one dimension | -| `sum_back(along=)` | `WindowSum` | a sum over a trailing window | +| File verb | Node | What the node names | +| ------------------ | ------------------- | ---------------------------------------------------- | +| `sum(over=)` | `Sum` | dims removed from the result | +| `sum(by=)` | `Sum` over a `Join` | a join, and the sum over the axes it opens | +| `at(by=)` | `Join` | a join whose groups are one row, with no sum over it | +| `shift(along=)` | `Translate` | a re-index along one dimension | +| `sum_back(along=)` | `WindowSum` | a sum over a trailing window | Nothing is abbreviated. diff --git a/docs/examples/operators.md b/docs/examples/operators.md index 1679d0fa..6f466621 100644 --- a/docs/examples/operators.md +++ b/docs/examples/operators.md @@ -75,7 +75,7 @@ $`\sum_{g \in \mathcal{G}} p_{t,g} \le \mathrm{limit}_{t} \qquad \forall\, t \in ```yaml description: >- - The membership reduction — `sum(array, by=relation, over=a, into=b)` lands the result on the + The membership reduction — `sum(array, by=relation, over=a, into=b)` groups the result by the column the relation is read to, which is what makes topology data rather than structure. @@ -112,8 +112,8 @@ $`\sum_{g \in \mathcal{G} \,:\, \mathrm{gen\_bus}(g) = b} p_{t,g} \le \mathrm{li ```yaml description: >- A call that names its ends — `sum(array, by=relation, over=a, into=b)` - consumes column `a` and lands on column `b`, and the other key column is - joined on, so each zone's total is taken per period. + joins on column `a` and groups by column `b`, and the other key column is + joined on and kept, so each zone's total is taken per period. dimensions: generator: { dtype: str } @@ -148,8 +148,8 @@ $`\sum_{g \in \mathcal{G} \,:\, \mathrm{zone\_of}(g,\ e) = z} p_{g,e} \ge \mathr ```yaml description: >- A call with several columns at each end — `sum(array, by=relation, over=[a, …], into=[b, …])` - consumes both key columns at once and lands on the product of both value - columns in one join. + joins on both key columns at once and groups by both value columns in one + join. dimensions: generator: { dtype: str } @@ -184,7 +184,7 @@ $`\sum_{g \in \mathcal{G},\ e \in \mathcal{E} \,:\, \mathrm{slot\_of.bus}(g,\ e) ```yaml description: >- - The adjoint of the membership reduction — `at(array, by=relation, over=a, into=b)` reads one + The same join with no group-by — `at(array, by=relation, over=a, into=b)` reads one coarse value once per fine label pointing at it. dimensions: diff --git a/docs/howto/declare-a-column.md b/docs/howto/declare-a-column.md index 12c8c173..38cd5a59 100644 --- a/docs/howto/declare-a-column.md +++ b/docs/howto/declare-a-column.md @@ -13,7 +13,7 @@ what the math does with the column. | The column… | is declared as | because | | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -| is an axis: something is indexed by it, or an aggregation lands terms on it | a `dimension` | its members are the coordinate set every table over it is reindexed onto | +| is an axis: something is indexed by it, or a grouping groups terms by it | a `dimension` | its members are the coordinate set every table over it is reindexed onto | | has one value per member of a dimension, or per tuple of several — a generator's bus, a line's two ends, a generator's zone by period | a `relation` with that `key` | it is a map every operator reads, and its values are checked against the dimensions they name | | relates members of two dimensions many-to-many, with nothing to weigh — which buses a generator may connect to | a bare `relation`, with no `values:` | `sum` reads it with both ends named, and a bare `where` tests it. Nothing reads it, because there is no one value to read | | relates members of two dimensions many-to-many, with a weight per pair — a link's efficiency to each bus, a cycle's lines | a `parameter` over both | the weight is the data, its row set is the relation, and the aggregation is `sum(w * x, over=a)` | diff --git a/docs/reference/language/expressions.md b/docs/reference/language/expressions.md index 61d03a99..b2978216 100644 --- a/docs/reference/language/expressions.md +++ b/docs/reference/language/expressions.md @@ -85,18 +85,18 @@ after a variable. The objective has no name at all. The dimension set of every expression is known before any data binds: -| Node | Dim set | Error | -| -------------------------------- | --------------------------------- | -------------------------------------------------------------------------------- | -| number | `{}` | | -| parameter / variable | its `dims` | | -| `-x`, `+x` | `dims(x)` | | -| `a + b`, `a * b`, `a / b` | `dims(a) ∪ dims(b)` | | -| `sum(x)` | `{}` | error if `dims(x)` is already empty | -| `sum(x, over=d)` | `dims(x) − {d}` | error if `d ∉ dims(x)` | -| `sum(x, by=l, over=a, into=b)` | `(dims(x) − consumed) ∪ produced` | the refusals under [how a relation is used](relations.md#how-a-relation-is-used) | -| `at(x, by=l, over=a, into=b)` | `(dims(x) − consumed) ∪ produced` | the same | -| `shift(x, along=d, offset=n)` | `dims(x)` | error if `d ∉ dims(x)` | -| `sum_back(x, along=d, window=n)` | `dims(x)` | error if `d ∉ dims(x)` | +| Node | Dim set | Error | +| -------------------------------- | ------------------------------ | -------------------------------------------------------------------------------- | +| number | `{}` | | +| parameter / variable | its `dims` | | +| `-x`, `+x` | `dims(x)` | | +| `a + b`, `a * b`, `a / b` | `dims(a) ∪ dims(b)` | | +| `sum(x)` | `{}` | error if `dims(x)` is already empty | +| `sum(x, over=d)` | `dims(x) − {d}` | error if `d ∉ dims(x)` | +| `sum(x, by=l, over=a, into=b)` | `(dims(x) − joined) ∪ grouped` | the refusals under [how a relation is used](relations.md#how-a-relation-is-used) | +| `at(x, by=l, over=a, into=b)` | `(dims(x) − joined) ∪ grouped` | the same | +| `shift(x, along=d, offset=n)` | `dims(x)` | error if `d ∉ dims(x)` | +| `sum_back(x, along=d, window=n)` | `dims(x)` | error if `d ∉ dims(x)` | A binary operator takes the **union** of the two dimension sets, so an outer product is allowed. The declaration's own dimensions are its **frame**, and a diff --git a/docs/reference/language/operators.md b/docs/reference/language/operators.md index eeec1f3d..870da5a1 100644 --- a/docs/reference/language/operators.md +++ b/docs/reference/language/operators.md @@ -15,7 +15,7 @@ in a reported expression, are all of them. A composition of them goes in | `sum(array)` | Every dimension that `array` carries collapses. The result is a scalar | | `sum(array, over=dim)` | `dim` collapses. `array` must carry `dim` | | `sum(array, by=relation, over=a, into=b)` | Column `a` collapses onto column `b`. The other key columns are joined on, so the array carries them and the result keeps them | -| `sum(array, by=relation, over=[a, …], into=[b, …])` | The same with several columns on either side: consumed together, landed on a product | +| `sum(array, by=relation, over=[a, …], into=[b, …])` | The same with several columns on either side: joined on together, grouped by a product | | `at(array, by=relation, over=a, into=b)` | Column `a` is replaced by column `b`, one value per coordinate. Either may be a list | | `shift(array, along=dim, offset=n)` | The value `n` positions earlier along `dim`. The vacated edge is **absent** | | `shift(array, along=dim, offset=n, edge='wrap')` | The value `n` positions earlier, counted cyclically, so nothing is vacated | @@ -41,8 +41,8 @@ result is a scalar. An operand that is already scalar, and an `over=` naming a dimension the operand does not carry, are both errors. -`sum(x, by=l, over=a, into=b)` sums through a [relation](relations.md), -consuming column `a` and landing the result on column `b`. A nodal balance is +`sum(x, by=l, over=a, into=b)` sums through a [relation](relations.md): it +joins on column `a` and groups by column `b`. A nodal balance is one `sum(by=)` per kind of component: ```yaml @@ -78,9 +78,10 @@ is null belongs to no group. ## `at` -`at(x, by=l, over=a, into=b)` reads the relation the other way. It consumes a -value column and produces the key, so it reads one coarse value once for each -fine label that points at it ([reads](relations.md#aggregates-and-reads)). +`at(x, by=l, over=a, into=b)` joins the relation the other way, with no +group-by. It joins on a value column and groups by the key, so every group is +one row, and it reads one coarse value once for each fine label that points at +it ([joins](relations.md#joins-and-group-bys)). `at` reads a variable as readily as a parameter. One decision taken per bus, read once by every line that touches the bus, is `at(decision, by=line_bus, over=bus, into=line)`. diff --git a/docs/reference/language/relations.md b/docs/reference/language/relations.md index 63ab8950..4e7480b7 100644 --- a/docs/reference/language/relations.md +++ b/docs/reference/language/relations.md @@ -72,15 +72,16 @@ column per declared column, named after it. ## How a relation is used -The declaration fixes no direction. A call names the columns it reads, and a -key column it names at neither end is **joined on**. +The declaration fixes no direction. A call **joins** the operand to the +relation on the columns they share, and **groups** what the join produces. The +call names the columns that decide both. -| kind | what it does | written as | -| --------- | ------------------------------------------- | ----------------------------------------------------- | -| aggregate | many rows of the operand collapse onto one | `sum(x, by=l, over=a, into=b)` | -| read | one row's value becomes a coordinate | `at(x, by=l, over=a, into=b)` | -| partition | the frame stays, and its rows are grouped | `shift`, `sum_back`, `position` with `by=l, within=c` | -| test | a row's presence keeps or cuts a coordinate | the relation's name in a `where` | +| kind | what it does | written as | +| ----------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------- | +| join and group by | rows of the operand match rows of the relation, and each group is added up | `sum(x, by=l, over=a, into=b)` | +| join | each row of the operand matches one row of the relation, and reads its value | `at(x, by=l, over=a, into=b)` | +| partition | the frame stays, and its rows are grouped | `shift`, `sum_back`, `position` with `by=l, within=c` | +| test | a row's presence keeps or cuts a coordinate | the relation's name in a `where` | Four rules hold for every use: @@ -90,37 +91,50 @@ Four rules hold for every use: 3. **The key is fixed.** To change it, declare a new relation. 4. **A dimension the relation does not name passes through** to the result. -### Aggregates and reads - -`over=` names the columns consumed and `into=` the columns produced, and either -may be a list. With `zone_of: { key: [generator, period], values: zone }`, the -bare `connection: { key: [generator, bus] }` and `p` over `[generator, period]`: - -| call | consumes | joins on | produces | result | -| -------------------------------------------------- | ----------- | ----------- | ----------- | --------------------- | -| `sum(p, by=zone_of, over=generator, into=zone)` | `generator` | `period` | `zone` | `[zone, period]` | -| `sum(p, by=zone_of, over=period, into=zone)` | `period` | `generator` | `zone` | `[generator, zone]` | -| `sum(p, by=connection, over=generator, into=bus)` | `generator` | nothing | `bus` | `[bus, period]` | -| `at(price, by=zone_of, over=zone, into=generator)` | `zone` | `period` | `generator` | `[generator, period]` | - -- **The result is the operand, less the consumed dimensions, plus the produced - ones.** The operand carries every dimension consumed or joined on, and none - that the call lands on. `sum(load * p, by=gen_bus, over=generator, into=bus)` - is refused; write `load * sum(p, by=gen_bus, over=generator, into=bus)`. -- **A read finds one row per coordinate, and a sum finds many.** The columns a - read lands on and joins on hold the whole key. A sum leaves a key column out, - and each call is refused in the other's case: +### Joins and group-bys + +Each column of the relation is either **joined on** or not, and either +**grouped by** or not. `over=` names the columns joined on and not grouped by. +`into=` names the columns grouped by and not joined on. Either may be a list. + +| column of the relation | joined on | grouped by | in the result | +| ----------------------------- | --------- | ---------- | ------------- | +| `over=` | yes | no | leaves | +| `into=` | no | yes | arrives | +| a key column the call omits | yes | yes | stays | +| a value column the call omits | no | no | is not read | + +With `zone_of: { key: [generator, period], values: zone }`, the bare +`connection: { key: [generator, bus] }` and `p` over `[generator, period]`: + +| call | joins on | groups by | result | +| -------------------------------------------------- | ------------------- | ------------------- | --------------------- | +| `sum(p, by=zone_of, over=generator, into=zone)` | `generator, period` | `zone, period` | `[zone, period]` | +| `sum(p, by=zone_of, over=period, into=zone)` | `period, generator` | `zone, generator` | `[generator, zone]` | +| `sum(p, by=connection, over=generator, into=bus)` | `generator` | `bus` | `[bus, period]` | +| `at(price, by=zone_of, over=zone, into=generator)` | `zone, period` | `generator, period` | `[generator, period]` | + +- **The result is the operand, less the columns joined on, plus the columns + grouped by.** The operand carries every column joined on, and none that the + call groups by: a column the operand carries would be joined on, not grouped + by. `sum(load * p, by=gen_bus, over=generator, into=bus)` is refused; write + `load * sum(p, by=gen_bus, over=generator, into=bus)`. +- **A sum adds up several rows per group, and a read finds one.** Where the + columns grouped by hold the whole key, every group is one row. That is a join + with no group-by, which is `at`. A `sum` there is refused toward `at`, and an + `at` whose groups hold several rows is refused toward `sum`: ```text - Constraint 'cap': sum(by=zone_of): this sum lands on the key ['generator', 'period'], so each coordinate has one term and nothing is added up — that is a read, which is at()'s. Write at(..., by=zone_of, over=['zone'], into=['generator']), or sum toward a value column. + Constraint 'cap': sum(by=zone_of): the columns this sum groups by, ['generator', 'period'], hold the whole key ['generator', 'period'], so every group is one row and nothing is added up — that is a join with no group-by, which is at()'s. Write at(..., by=zone_of, over=['zone'], into=['generator']), or sum toward a value column. ``` -- **A sum consumes at least one key column, and lands on any column it does not - consume.** Either end may name a value column beside a key one. `connection` - has only key columns, and the sum above consumes one and lands on the other. -- **A read consumes value columns, and lands on the key.** Its result carries - every key column — named in `into=`, or joined on — and whatever else the - operand carries that the read does not consume. +- **A sum joins on at least one key column, and groups by any column it does + not join on.** Either end may name a value column beside a key one. + `connection` has only key columns, and the sum above joins on one and groups + by the other. +- **A read joins on value columns, and groups by the key.** Its result carries + every key column, named in `into=` or left unnamed, and whatever else the + operand carries that the read does not join on. - **`over=` and `into=` name different columns**, and neither names two columns over one dimension. @@ -128,7 +142,8 @@ bare `connection: { key: [generator, bus] }` and `p` over `[generator, period]`: `shift(x, along=d, by=l, within=c)`, `sum_back(x, along=d, by=l, within=c)` and `position(d, by=l, within=c)` step along the key column over `d`, join on -the other key columns, and group by the value columns `within=` names. The +the other key columns, and partition the rows by the value columns `within=` +names. The frame does not change. `within=` is written whenever `by=` is. It may name two columns over one dimension, may not name a key column, and a bare relation partitions nothing. diff --git a/docs/reference/notation.md b/docs/reference/notation.md index 3452684a..96f37fc5 100644 --- a/docs/reference/notation.md +++ b/docs/reference/notation.md @@ -52,7 +52,7 @@ relations: zone_of: { key: bus, values: zone } area_of: { key: bus, values: zone } # a second map into the same set, to compare against season_of: { key: snapshot, values: season } - gen_zone: { key: [generator, snapshot], values: zone } # a map keyed by two dimensions: a call consumes one and joins on the other + gen_zone: { key: [generator, snapshot], values: zone } # a map keyed by two dimensions: a call sums one away and joins on the other rep_of: { key: snapshot, values: { rep: snapshot } } # a map into its own dimension: the representative snapshot connection: { key: [generator, bus] } # a bare relation, with no value columns: many-to-many, read only by sum with both ends named gen_bt: { key: generator, values: [bus, technology] } # one table with two value columns, read to both at once @@ -381,12 +381,12 @@ seasonal_window: \sum_{t' \in \mathcal{T} \,:\, 0 \le t -^{\mathrm{season\_of}(t)} t' < 3} \mathit{on}_{t',g} \le \mathit{units}_{g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} ``` -#### `pullback` +#### `lookup` at(), which re-indexes through a relation instead of an offset ```yaml -pullback: +lookup: dims: [snapshot, bus] expression: spill <= at(zone_cap, by=zone_of, over=zone, into=bus) ``` @@ -409,12 +409,12 @@ grouped_once: \sum_{g \in \mathcal{G} \,:\, \mathrm{gen\_bt.bus}(g) = b \wedge \mathrm{gen\_bt.technology}(g) = e} p_{t,g} \le \mathrm{tech\_cap}_{b,e} \qquad \forall\, t \in \mathcal{T},\ b \in \mathcal{B},\ e \in \mathcal{E} ``` -#### `pulled_back_once` +#### `looked_up_once` -its adjoint, reading one slot through two columns of one table +the same table joined the other way, reading one slot through two columns ```yaml -pulled_back_once: +looked_up_once: dims: [generator] expression: units <= at(tech_cap, by=gen_bt, over=[bus, technology], into=generator) ``` @@ -483,7 +483,7 @@ representative: #### `zonal` -a grouping through a two-key map, consuming one key: the condition reads the other, and the row keeps it +a grouping through a two-key map, summing one key away: the condition reads the other, and the row keeps it ```yaml zonal: @@ -497,7 +497,7 @@ zonal: #### `zonal_history` -the same table consuming its other key +the same table summing its other key away ```yaml zonal_history: @@ -523,12 +523,12 @@ zonal_membership: \sum_{g \in \mathcal{G} \,:\, \mathrm{gen\_zone}(g,\ t) \text{ is defined}} \mathit{units}_{g} \le \mathrm{budget} \qquad \forall\, t \in \mathcal{T} ``` -#### `zonal_pullback` +#### `zonal_lookup` -its adjoint, reading the slot the row's own snapshot puts the generator in +the same table joined the other way, reading the slot the row's own snapshot puts the generator in ```yaml -zonal_pullback: +zonal_lookup: dims: [snapshot, generator] where: "gen_zone == 'north' AND position(generator, by=gen_zone, within=zone) == 0" expression: p <= at(spill * zone_cap, by=gen_zone, into=generator, over=zone) diff --git a/docs/reference/reading.md b/docs/reference/reading.md index 14af2af1..539db3a5 100644 --- a/docs/reference/reading.md +++ b/docs/reference/reading.md @@ -164,7 +164,7 @@ Three predicates read another predicate rather than a declaration. A `CountComparison` carries the mask it counts and the dimension it counts away. A `TranslatedPredicate` carries the mask it reads at a neighbouring coordinate. A `PulledBackPredicate` carries the mask it reads through a -relation, and the `Direction` it reads in. Each holds that mask as a `Mask`, +relation, and the `JoinColumns` it joins on and groups by. Each holds that mask as a `Mask`, where a connective holds a bare predicate: the walk recurses through a connective and stops at these, so read the field where you need what is inside. `.names_read` and `.dims` already see through all three, and the @@ -213,7 +213,7 @@ Every declared axis has an entry. A coupling that a `piecewise:` expansion introduced is named under the declaration the expansion emitted. - `coupled` names each declaration that ties the whole axis together: a sum - over the axis in a constraint, a grouping that consumes the axis, a wrapped + over the axis in a constraint, a grouping that sums the axis away, a wrapped shift, or a set. After the dash, each entry names the one change that would remove the tie. - `undecided` lists each read whose reach only the data can say, as a `Reach`: diff --git a/examples/operators/at.yaml b/examples/operators/at.yaml index 7d5eb2d2..3cf8adbe 100644 --- a/examples/operators/at.yaml +++ b/examples/operators/at.yaml @@ -3,7 +3,7 @@ # SPDX-License-Identifier: MIT description: >- - The adjoint of the membership reduction — `at(array, by=relation, over=a, into=b)` reads one + The same join with no group-by — `at(array, by=relation, over=a, into=b)` reads one coarse value once per fine label pointing at it. dimensions: diff --git a/examples/operators/sum_by.yaml b/examples/operators/sum_by.yaml index e02cf26c..4eb1eebd 100644 --- a/examples/operators/sum_by.yaml +++ b/examples/operators/sum_by.yaml @@ -3,7 +3,7 @@ # SPDX-License-Identifier: MIT description: >- - The membership reduction — `sum(array, by=relation, over=a, into=b)` lands the result on the + The membership reduction — `sum(array, by=relation, over=a, into=b)` groups the result by the column the relation is read to, which is what makes topology data rather than structure. diff --git a/examples/operators/sum_by_column_lists.yaml b/examples/operators/sum_by_column_lists.yaml index 98f70b7f..787ac2ac 100644 --- a/examples/operators/sum_by_column_lists.yaml +++ b/examples/operators/sum_by_column_lists.yaml @@ -4,8 +4,8 @@ description: >- A call with several columns at each end — `sum(array, by=relation, over=[a, …], into=[b, …])` - consumes both key columns at once and lands on the product of both value - columns in one join. + joins on both key columns at once and groups by both value columns in one + join. dimensions: generator: { dtype: str } diff --git a/examples/operators/sum_by_columns.yaml b/examples/operators/sum_by_columns.yaml index 683fcd3c..8c9fc3fd 100644 --- a/examples/operators/sum_by_columns.yaml +++ b/examples/operators/sum_by_columns.yaml @@ -4,8 +4,8 @@ description: >- A call that names its ends — `sum(array, by=relation, over=a, into=b)` - consumes column `a` and lands on column `b`, and the other key column is - joined on, so each zone's total is taken per period. + joins on column `a` and groups by column `b`, and the other key column is + joined on and kept, so each zone's total is taken per period. dimensions: generator: { dtype: str } diff --git a/mkdocs.yml b/mkdocs.yml index 5fc61277..6075b522 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -76,6 +76,7 @@ nav: - The limits: about/limits.md - What counts as language: about/what-counts-as-language.md - What counts as public API: about/what-counts-as-public-api.md + - Relations as linear maps: about/relations-as-linear-maps.md - Contributing: contributing.md - Changelog: CHANGELOG.md diff --git a/schema/math-spec.schema.json b/schema/math-spec.schema.json index 7dec94e3..bf2ce5de 100644 --- a/schema/math-spec.schema.json +++ b/schema/math-spec.schema.json @@ -496,7 +496,7 @@ }, "RelationBlock": { "additionalProperties": false, - "description": "A named relation between dimensions: the columns a row is keyed by, and the columns that key determines.\n\nEach side is a dimension, a list of them, or a mapping of column name to\ndimension where two columns share one. ``key:`` is the claim the language\nchecks at bind: one row per key tuple, so every ``values:`` column is a\nfunction of it. A relation with no ``values:`` is **bare** \u2014 every column is\nin its key, a row is its own identity, and nothing reads it::\n\n relations:\n gen_bus: {key: generator, values: bus}\n gen_bt: {key: [generator], values: [bus, technology]}\n zone_of: {key: [generator, period], values: zone}\n ends: {key: line, values: {bus0: bus, bus1: bus}}\n connection: {key: [generator, bus]}\n\nAn operator reads the table in the direction the call names\n(``over=``, ``into=``), joining on the other key columns; the\ndeclaration fixes no direction. The map itself is data, and arrives at bind\ntime under the relation's name, one column per role.", + "description": "A named relation between dimensions: the columns a row is keyed by, and the columns that key determines.\n\nEach side is a dimension, a list of them, or a mapping of column name to\ndimension where two columns share one. ``key:`` is the claim the language\nchecks at bind: one row per key tuple, so every ``values:`` column is a\nfunction of it. A relation with no ``values:`` is **bare** \u2014 every column is\nin its key, a row is its own identity, and nothing reads it::\n\n relations:\n gen_bus: {key: generator, values: bus}\n gen_bt: {key: [generator], values: [bus, technology]}\n zone_of: {key: [generator, period], values: zone}\n ends: {key: line, values: {bus0: bus, bus1: bus}}\n connection: {key: [generator, bus]}\n\nAn operator joins the table on the columns ``over=`` names and every\nother key column, and groups by the columns ``into=`` names; the\ndeclaration fixes no direction. The map itself is data, and arrives at bind\ntime under the relation's name, one column per role.", "properties": { "description": { "anyOf": [ diff --git a/src/math_spec/advice.py b/src/math_spec/advice.py index 2d77d0b0..8712b0a1 100644 --- a/src/math_spec/advice.py +++ b/src/math_spec/advice.py @@ -14,7 +14,7 @@ from math_spec.boundedness import unbounded_notes from math_spec.errors import Advice from math_spec.lowering import to_program -from math_spec.program import GroupSum, Pullback, walk +from math_spec.program import Join, walk if TYPE_CHECKING: from collections.abc import Mapping @@ -51,7 +51,7 @@ def _never_an_axis(program: Program) -> list[Advice]: reached: set[str] = set() for declaration in (*program.parameters.values(), *program.variables.values(), *program.constraints.values()): reached.update(declaration.dims) - reached |= _produced_axes(program) + reached |= _grouped_axes(program) reached |= {dim for lk in program.relations.values() for dim in lk.dims} return [ @@ -67,14 +67,14 @@ def _never_an_axis(program: Program) -> list[Advice]: ] -def _produced_axes(program: Program) -> set[str]: +def _grouped_axes(program: Program) -> set[str]: """The axes the expressions create beyond what any declaration indexes. - ``sum(by=)`` lands on its target and ``at()`` spreads onto its fine - dimension: either way, the dims the direction produces. + ``sum(by=)`` groups onto its target and ``at()`` spreads onto its fine + dimension: either way, the dims the join groups by and did not join on. """ axes: set[str] = set() for node in walk(*program.roots): - if isinstance(node, GroupSum | Pullback): - axes.update(node.direction.produced_dims) + if isinstance(node, Join): + axes.update(node.columns.added_dims) return axes diff --git a/src/math_spec/boundedness.py b/src/math_spec/boundedness.py index 1102abe8..89d8cb03 100644 --- a/src/math_spec/boundedness.py +++ b/src/math_spec/boundedness.py @@ -24,12 +24,11 @@ Divide, Dual, Expression, - GroupSum, + Join, Multiply, Negate, Parameter, Power, - Pullback, Sum, Translate, Variable, @@ -162,7 +161,7 @@ def _record_signs(node: Expression, sign: Sign, signs: dict[str, Sign]) -> None: _record_signs(node.base, None, signs) _record_signs(node.exponent, None, signs) return - if isinstance(node, Sum | GroupSum | Pullback | Translate | WindowSum | Cases): + if isinstance(node, Sum | Join | Translate | WindowSum | Cases): for child in children(node): _record_signs(child, sign, signs) return diff --git a/src/math_spec/degree.py b/src/math_spec/degree.py index facc5faa..ac840f8c 100644 --- a/src/math_spec/degree.py +++ b/src/math_spec/degree.py @@ -29,7 +29,6 @@ Divide, Dual, Expression, - GroupSum, Multiply, Power, Sum, @@ -175,10 +174,10 @@ def _multi_term(node: Expression) -> bool: def _joins_terms(node: Expression) -> bool: """Whether *node* itself makes several terms of one: a reduction over a variable, or a sum of two variable-carrying sides. - A pullback and a translation re-index and are not reductions: they move a + A lookup and a translation re-index and are not reductions: they move a term, leaving one term where there was one. """ - if isinstance(node, Sum | GroupSum | WindowSum): + if isinstance(node, Sum | WindowSum): return carries_variable(node.operand) return isinstance(node, Add) and carries_variable(node.left) and carries_variable(node.right) diff --git a/src/math_spec/dimensions.py b/src/math_spec/dimensions.py index e421f93e..d0bd86dc 100644 --- a/src/math_spec/dimensions.py +++ b/src/math_spec/dimensions.py @@ -23,12 +23,12 @@ CountComparison, DimensionComparison, DimensionPosition, - Direction, Divide, Dual, Expression, ExpressionComparison, - GroupSum, + Join, + JoinColumns, Mask, Multiply, Named, @@ -38,7 +38,6 @@ ParameterDefined, Partition, Power, - Pullback, PulledBackPredicate, RelationComparison, RelationDefined, @@ -87,10 +86,8 @@ def dims_of(node: Expression, schema: Spec, context: str) -> frozenset[str]: inner = dims_of(node.operand, schema, context) if isinstance(node, Sum): return _sum_dims(node, inner, context) - if isinstance(node, GroupSum): - return _group_sum_dims(node, inner, context) - if isinstance(node, Pullback): - return _at_dims(node, inner, context) + if isinstance(node, Join): + return join_dims(node.columns, inner, context, 'the expression') if isinstance(node, Translate | WindowSum): return _translation_dims(node, inner, schema, context) @@ -115,47 +112,47 @@ def _not_carried(context: str, call: str, inner: frozenset[str], rewrite: str) - def _sum_dims(node: Sum, inner: frozenset[str], context: str) -> frozenset[str]: """``sum`` reduces each named dim away, so the operand carries every one.""" - for consumed in node.over: - if consumed not in inner: - raise DimensionError(_not_carried(context, f'sum(over={consumed})', inner, 'drop the sum, or fix the dim')) + for summed in node.over: + if summed not in inner: + raise DimensionError(_not_carried(context, f'sum(over={summed})', inner, 'drop the sum, or fix the dim')) return inner - frozenset(node.over) -def _group_sum_dims(node: GroupSum, inner: frozenset[str], context: str) -> frozenset[str]: - """``sum`` through a relation: the consumed dim goes, the produced dims arrive, the joined stay.""" - direction = node.direction - if missing := sorted(set(direction.consumed_dims) - inner): - raise DimensionError( - _not_carried( - context, - f'sum(by={direction.name}) consumes {missing}, the dims it reads from,', - inner, - 'drop the sum, or fix the dim', - ) - ) - return _read_dims(f'sum(by={direction.name})', direction, inner, context) - - -def _at_dims(node: Pullback, inner: frozenset[str], context: str) -> frozenset[str]: - """``at`` is the adjoint of ``sum(by=)``: it consumes the dims a sum produces and produces the ones it consumes.""" - return pulled_back_dims(node.direction, inner, context, 'the expression') +def join_dims(columns: JoinColumns, inner: frozenset[str], context: str, operand: str) -> frozenset[str]: + """The dims *inner* has once *columns* joins it, an expression's or a predicate's alike. - -def pulled_back_dims(direction: Direction, inner: frozenset[str], context: str, operand: str) -> frozenset[str]: - """The dims *inner* has once ``at`` reads it through *direction*, an expression's or a predicate's alike. + The dims joined on go, the dims grouped by arrive, and each column joined + on and not grouped by opens its own axis (:attr:`JoinColumns.axes`), which + the :class:`~math_spec.program.Sum` over the join takes away. A column + both joined on and grouped by keeps its dim. The call a refusal quotes is + ``at`` where each group is one row, and ``sum`` otherwise. Raises: - DimensionError: *operand* does not carry a dim the read consumes or - joins on, or already carries one it lands on. + DimensionError: *operand* does not carry a dim the call joins on, or + already carries one the call adds. """ - if absent := sorted(set(direction.consumed_dims) - inner): + lookup = columns.one_row_per_group + call = f'{"at" if lookup else "sum"}(by={columns.name})' + if missing := sorted(set(columns.dropped_dims) - inner): + if lookup: + raise DimensionError( + f'{context}: {call} joins on {missing}, which {operand} does not carry (dims ' + f'{sorted(inner)}). A lookup joins the operand on the columns it reads at — ' + f'sum is the call that groups by them.' + ) + raise DimensionError( + _not_carried(context, f'{call} joins on {missing} to sum it away,', inner, 'drop the sum, or fix the dim') + ) + added, dropped = set(columns.added_dims), set(columns.dropped_dims) + if clash := sorted((added & inner) - dropped): raise DimensionError( - f'{context}: at(by={direction.name}) reads through ' - f'{absent}, which {operand} does not carry (dims ' - f'{sorted(inner)}). A pullback needs the coarse dims to read *from* — ' - f'sum is the direction that produces them.' + f'{context}: {call} groups by {clash}, which the expression already carries.\n' + f'A join on a column the operand carries matches it rather than grouping by it, so a ' + f'call brings the dims it groups by. Move the factor carrying {clash} outside the operator, ' + f'or group by a column over another dimension.' ) - return _read_dims(f'at(by={direction.name})', direction, inner, context) + _check_joined(call, columns, inner, context) + return (inner - set(columns.joined_dims)) | set(columns.grouped_dims) | set(columns.axes) #: The verb a file writes each translation with, which its refusals quote. @@ -180,48 +177,26 @@ def _translation_dims(node: Translate | WindowSum, inner: frozenset[str], schema return inner -def _read_dims(call: str, direction: Direction, inner: frozenset[str], context: str) -> frozenset[str]: - """The dims after a relation is read in *direction*: the consumed go, the produced arrive, the joined stay. - - The dims a call lands on are its own to bring, so the operand does not - already carry one. Where it does, the call would tie the operand's axis to - the one it produces rather than adding it, and it reads the same either - way. A relation into its own dimension is not that case: there the dim - landed on is the dim just consumed, so every factor is read at the - coordinate the sum runs over, and nothing is tied. - """ - consumed, produced = set(direction.consumed_dims), set(direction.produced_dims) - if clash := sorted((produced & inner) - consumed): - raise DimensionError( - f'{context}: {call} lands on {clash}, which the expression already carries.\n' - f'A call brings the dims it lands on, so that reading it tells you what it ' - f'adds. Move the factor carrying {clash} outside the operator, or read to a column ' - f'over another dimension.' - ) - _check_joined(call, direction, inner, context) - return (inner - consumed) | produced - - -def _check_joined(call: str, use: Direction | Partition, inner: frozenset[str], context: str) -> None: - """The columns a call joins on are read at their dimensions, so the operand carries every one, each once. +def _check_joined(call: str, use: JoinColumns | Partition, inner: frozenset[str], context: str) -> None: + """The columns a call joins on are matched at their dimensions, so the operand carries every one, each once. - A joined dimension the call also consumes is the same ambiguity as two - joined columns over one dimension: the operand's one coordinate would - have to be read as both. A partition consumes nothing. + Two joined columns over one dimension would match the operand's one + coordinate twice: the ``over=`` column and an unnamed key column, or two + unnamed key columns. A partition joins on the key columns it does not + step along. """ dims = use.joined_dims if missing := sorted(set(dims) - inner): raise DimensionError( f'{context}: {call} joins on {missing} (columns {[r for r in use.joined if use.dim(r) in missing]} ' - f"of '{use.name}'), which the expression does not carry (dims {sorted(inner)}). A relation is " - f'read between two of its columns and joined at the others — index the operand by them, or ' - f'read it between different columns.' + f"of '{use.name}'), which the expression does not carry (dims {sorted(inner)}). A join matches " + f'the operand on every key column the call does not name — index the operand by them, or ' + f'name them in the call.' ) - consumed = use.consumed_dims if isinstance(use, Direction) else () - if twice := sorted({d for d in dims if dims.count(d) > 1 or d in consumed}): + if twice := sorted({d for d in dims if dims.count(d) > 1}): raise DimensionError( f"{context}: {call} joins '{use.name}' on {twice} through more than one column, and the operand " - f'carries each dimension once. Read between different columns, or use a relation whose joined ' + f'carries each dimension once. Join on distinct dimensions, or use a relation whose key ' f'columns are over distinct dimensions.' ) @@ -243,7 +218,7 @@ def _check_named_amount( f"— declare '{amount}' over dims '{node.along}' is not one of." ) groups = ( - frozenset(node.partition.dim(v) for v in node.partition.group) if node.partition is not None else frozenset() + frozenset(node.partition.dim(v) for v in node.partition.grouped) if node.partition is not None else frozenset() ) if stray := sorted(frozenset(declared.dims) - inner - groups): raise DimensionError( @@ -367,7 +342,7 @@ def _check_where_dims( case TranslatedPredicate(): leaf = f"a where-predicate translated along '{atom.along}'" case PulledBackPredicate(): - leaf = f"a where-predicate read through '{atom.direction.name}'" + leaf = f"a where-predicate read through '{atom.columns.name}'" case _: assert_never(atom) raise DimensionError( diff --git a/src/math_spec/exclusivity.py b/src/math_spec/exclusivity.py index bd9cb857..71c3a199 100644 --- a/src/math_spec/exclusivity.py +++ b/src/math_spec/exclusivity.py @@ -285,7 +285,7 @@ def _observe( raise Undecidable(msg) if isinstance(node, PulledBackPredicate): msg = ( - f"it reads a predicate through '{node.direction.name}', and which rows that admits only the data " + f"it reads a predicate through '{node.columns.name}', and which rows that admits only the data " 'decides — test this row, or precompute the read as a boolean parameter and test that' ) raise Undecidable(msg) @@ -321,7 +321,7 @@ def _subject_of(node: TypedPredicate) -> Subject: case DimensionPosition(name=name, partition=partition): if partition is None: return Subject('rank', name) - return Subject('rank', name, partition.name, partition.group) + return Subject('rank', name, partition.name, partition.grouped) case RelationDefined(name=name) | RelationComparison(name=name): return Subject('relation', name) case RelationPairComparison(name=name, other=other): diff --git a/src/math_spec/lowering.py b/src/math_spec/lowering.py index 61aa81ca..ea587336 100644 --- a/src/math_spec/lowering.py +++ b/src/math_spec/lowering.py @@ -160,7 +160,7 @@ def inline(node: program.Expression | program.Named) -> program.Expression: return program.Power(inline(node.base), inline(node.exponent)) if isinstance(node, program.Divide): return program.Divide(inline(node.numerator), inline(node.divisor)) - if isinstance(node, program.Sum | program.GroupSum | program.Pullback | program.Translate | program.WindowSum): + if isinstance(node, program.Sum | program.Join | program.Translate | program.WindowSum): return replace(node, operand=inline(node.operand)) if isinstance(node, program.Cases): return program.Cases(tuple(program.Region(inline_mask(r.when), inline(r.value)) for r in node.regions)) diff --git a/src/math_spec/model.py b/src/math_spec/model.py index e2b08e14..45a7ce30 100644 --- a/src/math_spec/model.py +++ b/src/math_spec/model.py @@ -189,8 +189,8 @@ class RelationBlock(_StrictBlock): ends: {key: line, values: {bus0: bus, bus1: bus}} connection: {key: [generator, bus]} - An operator reads the table in the direction the call names - (``over=``, ``into=``), joining on the other key columns; the + An operator joins the table on the columns ``over=`` names and every + other key column, and groups by the columns ``into=`` names; the declaration fixes no direction. The map itself is data, and arrives at bind time under the relation's name, one column per role. """ diff --git a/src/math_spec/operators.py b/src/math_spec/operators.py index cfa31e9e..d8249694 100644 --- a/src/math_spec/operators.py +++ b/src/math_spec/operators.py @@ -40,11 +40,11 @@ class Builtin: dimension_kwargs: tuple[str, ...] = () relation_kwargs: tuple[str, ...] = () #: Kwargs naming a column of the relation ``by=`` names — ``over=`` and - #: ``into=`` — which resolution folds into the direction it is read in. + #: ``into=`` — which resolution folds into the join the call makes. role_kwargs: tuple[str, ...] = () #: Kwargs naming a dimension on their own and a column of the relation where #: ``by=`` names one. ``sum(x, over=generator)`` reduces the dimension - #: away; ``sum(x, by=l, over=c)`` names the column the call consumes. + #: away; ``sum(x, by=l, over=c)`` names the column the call joins on and sums away. #: One meaning — what leaves the frame — read in the namespace ``by=`` #: decides. dimension_or_role_kwargs: tuple[str, ...] = () @@ -93,7 +93,7 @@ def kind_of( #: The closed operator set. ``by=`` is the one keyword that addresses a relation, #: and a relation carries its own dimensions, so no sibling kwarg restates them. #: On ``shift`` and ``sum_back`` it partitions the axis the operator steps along: it -#: says which rows are neighbours, not which group a term lands in, and +#: says which rows are neighbours, not which group a term is added to, and #: ``within=`` names the value columns the group is made of, on every call #: that names a ``by=``. BUILTINS: dict[str, Builtin] = { diff --git a/src/math_spec/program.py b/src/math_spec/program.py index b55f0232..21d63622 100644 --- a/src/math_spec/program.py +++ b/src/math_spec/program.py @@ -52,7 +52,6 @@ 'DimensionDeclaration', 'DimensionDtype', 'DimensionPosition', - 'Direction', 'Divide', 'Dual', 'Expression', @@ -60,7 +59,8 @@ 'ExpressionDeclaration', 'FanIn', 'Footprint', - 'GroupSum', + 'Join', + 'JoinColumns', 'Mask', 'Multiply', 'Named', @@ -80,7 +80,6 @@ 'Predicate', 'PredicateOperator', 'Program', - 'Pullback', 'PulledBackPredicate', 'QuadraticPosition', 'Reach', @@ -235,37 +234,38 @@ class Divide: @dataclass(frozen=True) class Sum: - """Sum ``operand`` over the named dims, removing them from the result.""" + """Sum ``operand`` over the named dims, removing them from the result. - operand: Expression - over: tuple[str, ...] - - -@dataclass(frozen=True) -class GroupSum: - """Sum ``operand`` through a relation: the dims ``direction`` consumes go, the dims it produces arrive, the dims it joins on stay. - - The join keys on the consumed columns and every joined column, and the - operand carries every dim consumed or joined on. + A name in ``over`` is a dimension, or one of the axes a :class:`Join` + under it opens (:attr:`JoinColumns.axes`): ``sum(x, by=relation, over=a, + into=b)`` lowers to a ``Sum`` over a ``Join``, over the axis of each column + the join does not group by. The join is the join, and this node is the + group-by that follows it. """ operand: Expression - direction: Direction + over: tuple[str, ...] @dataclass(frozen=True) -class Pullback: - """Read ``operand`` through a relation — the adjoint of :class:`GroupSum`. - - The dims ``direction`` consumes go and the dims it produces arrive, one - value per coordinate because the read takes value columns at a key the - result fixes, which the loader checks. The join fans out, many - produced tuples sharing one consumed tuple — at each coordinate of the - joined columns, which the operand carries and the result keeps. +class Join: + """Join ``operand`` to a relation on the columns ``columns`` joins on, one row per matching row of the relation. + + The operand carries every dim joined on. The result has the operand's + dims, less the dims joined on, plus the dims grouped by, plus one axis + per column joined on and not grouped by (:attr:`JoinColumns.axes`), named + for the relation's column and not for its dimension. So a column dropped + and a column added over one dimension stay two axes. A :class:`Sum` over + those axes is the group-by that follows the join, which is how + ``sum(x, by=relation, over=a, into=b)`` lowers. Where the grouped columns + hold the relation's whole key, they determine every other column, the + join opens no axis, and the bare ``Join`` is ``at(x, by=relation, + over=a, into=b)``: one value per row of the result, fanned out where + several key tuples share the values joined on. """ operand: Expression - direction: Direction + columns: JoinColumns @dataclass(frozen=True) @@ -384,8 +384,7 @@ class Named: | Power | Divide | Sum - | GroupSum - | Pullback + | Join | Translate | WindowSum | Cases @@ -402,13 +401,13 @@ def fan_in(expression: Expression | Named) -> FanIn: """ if isinstance(expression, Named): return fan_in(expression.body) - if isinstance(expression, (Sum, GroupSum)): + if isinstance(expression, Sum): return 'many-to-one' if isinstance(expression, WindowSum): return 'one-to-many' if isinstance( expression, - (Constant, Parameter, Variable, Dual, Negate, Add, Multiply, Power, Divide, Pullback, Translate, Cases), + (Constant, Parameter, Variable, Dual, Negate, Add, Multiply, Power, Divide, Join, Translate, Cases), ): return 'one-to-one' assert_never(expression) @@ -426,7 +425,7 @@ def children(expression: Expression) -> tuple[Expression, ...]: return (expression.numerator, expression.divisor) if isinstance(expression, Power): return (expression.base, expression.exponent) - if isinstance(expression, (Sum, GroupSum, Pullback, Translate, WindowSum)): + if isinstance(expression, (Sum, Join, Translate, WindowSum)): return (expression.operand,) if isinstance(expression, Cases): return tuple(region.value for region in expression.regions) @@ -480,59 +479,105 @@ def dim(self, role: str) -> str: @dataclass(frozen=True) -class Direction: - """One relation as one call reads it — which columns are consumed, which produced, which joined on. +class JoinColumns: + """One relation as one call joins it: the columns joined on, and the columns grouped by. The declaration fixes no direction; the call does, and this is the one it named. ``name`` is the relation's, as :attr:`Program.relations` keys it. - ``consumed``, ``produced`` and ``joined`` are *roles* — column names of - ``relation``, which binds every role to its dimension and names the key. - ``joined`` is the key roles the call did not name (every role, for a bare - relation): the join keys on them, and a value role left unnamed is not - read. + ``joined`` and ``grouped`` are *roles* — column names of ``relation``, + which binds every role to its dimension and names the key. ``joined`` is + every column the join matches the operand on: the ``over=`` columns, then + every key column the call did not name. ``grouped`` is every column of the + relation the result keeps: the ``into=`` columns, then the same unnamed + key columns. A column in neither is not read, so a relation may gain a + value column without changing what a call means. + + A column both joined on and grouped by stays in the frame. One joined on + and not grouped by is :attr:`dropped`, and one grouped by and not joined + on is :attr:`added`, so the frame after the call is the operand's dims, + less the dims joined on, plus the dims grouped by. """ name: str relation: RelationDeclaration - consumed: tuple[str, ...] - produced: tuple[str, ...] joined: tuple[str, ...] + grouped: tuple[str, ...] def dim(self, role: str) -> str: """The dimension *role* is bound to.""" return self.relation.dim(role) @property - def consumed_dims(self) -> tuple[str, ...]: - return tuple(self.dim(role) for role in self.consumed) + def dropped(self) -> tuple[str, ...]: + """The roles joined on and not grouped by: what the call sums away or looks up.""" + return tuple(role for role in self.joined if role not in self.grouped) + + @property + def added(self) -> tuple[str, ...]: + """The roles grouped by and not joined on: what the call brings into the frame.""" + return tuple(role for role in self.grouped if role not in self.joined) @property - def produced_dims(self) -> tuple[str, ...]: - return tuple(self.dim(role) for role in self.produced) + def kept(self) -> tuple[str, ...]: + """The roles both joined on and grouped by: the key columns the call did not name.""" + return tuple(role for role in self.joined if role in self.grouped) @property def joined_dims(self) -> tuple[str, ...]: return tuple(self.dim(role) for role in self.joined) + @property + def grouped_dims(self) -> tuple[str, ...]: + return tuple(self.dim(role) for role in self.grouped) + + @property + def dropped_dims(self) -> tuple[str, ...]: + return tuple(self.dim(role) for role in self.dropped) + + @property + def added_dims(self) -> tuple[str, ...]: + return tuple(self.dim(role) for role in self.added) + + @property + def kept_dims(self) -> tuple[str, ...]: + return tuple(self.dim(role) for role in self.kept) + + @property + def one_row_per_group(self) -> bool: + """Whether the grouped columns hold the relation's whole key, so each group is one row: a lookup, not a sum.""" + return set(self.relation.key) <= set(self.grouped) + + @property + def axes(self) -> tuple[str, ...]: + """The axis the join opens for each column it drops, ``relation.column``, empty where each group is one row. + + A dimension's name holds no dot, so an axis never meets one. A column + the grouped columns determine opens none: in a lookup they hold the + key, and the key determines every column. + """ + if self.one_row_per_group: + return () + return tuple(f'{self.name}.{role}' for role in self.dropped) + @dataclass(frozen=True) class Partition: - """One relation as a partition steps along it — the key column stepped along, the group columns, and the key columns joined on. + """One relation as a partition steps along it — the key column stepped along, the columns partitioned by, and the key columns joined on. ``name`` is the relation's, as :attr:`Program.relations` keys it. - ``along``, ``group`` and ``joined`` are *roles* — column names of + ``along``, ``grouped`` and ``joined`` are *roles* — column names of ``relation``, which binds every role to its dimension and names the key. ``along`` is the one key column over the dimension stepped along, and - the frame keeps it. ``group`` is the value columns ``within=`` named, - read at the row's key. ``joined`` is the other key columns, whose - dimensions the frame carries. Nothing is consumed and nothing is - produced: the frame does not change. + the frame keeps it. ``grouped`` is the value columns ``within=`` named, + read at the row's key: the partition's group. ``joined`` is the other key + columns, whose dimensions the frame carries. No column is dropped and + none is added: the frame does not change. """ name: str relation: RelationDeclaration along: str - group: tuple[str, ...] + grouped: tuple[str, ...] joined: tuple[str, ...] def dim(self, role: str) -> str: @@ -755,7 +800,7 @@ class Separability: is pointwise; a ``shift`` of ``-2`` is ``2``. coupled: Each declaration that ties the axis together, to what ties it and the one modelling change that would not: a sum over the axis - in a constraint, a grouping that consumes it, a wrapped + in a constraint, a grouping that sums it away, a wrapped translation, a set. No window satisfies these, and no rewrite here would keep the model's meaning, so the remedy is named rather than applied. @@ -1191,12 +1236,13 @@ class PulledBackPredicate: True at a coordinate where the relation has a row and *operand* holds at the coordinate that row reads. False where the relation has no row, which - is what a missing row already means in a mask. The dims ``direction`` - consumes go and the dims it produces arrive, as :class:`Pullback`'s do. + is what a missing row already means in a mask. The dims ``columns`` + joins on go and the dims it groups by arrive, as a lookup + :class:`Join`'s do. """ operand: Mask - direction: Direction + columns: JoinColumns dims: tuple[str, ...] @@ -1336,7 +1382,7 @@ def _atom_names(atom: TypedPredicate) -> frozenset[str]: case TranslatedPredicate(): return atom.operand.names_read case PulledBackPredicate(): - return atom.operand.names_read | {atom.direction.name} + return atom.operand.names_read | {atom.columns.name} case DimensionComparison() | DimensionPosition(): return frozenset() case _: @@ -1356,8 +1402,8 @@ def _names_under(*expressions: Expression) -> frozenset[str]: for node in walk(*expressions): if isinstance(node, Cases): names.update(*(region.when.names_read for region in node.regions)) - elif isinstance(node, (GroupSum, Pullback)): - names.add(node.direction.name) + elif isinstance(node, Join): + names.add(node.columns.name) elif isinstance(node, (Translate, WindowSum)): if node.partition is not None: names.add(node.partition.name) diff --git a/src/math_spec/resolution.py b/src/math_spec/resolution.py index 16ff529d..c67b146a 100644 --- a/src/math_spec/resolution.py +++ b/src/math_spec/resolution.py @@ -43,7 +43,7 @@ nested, parse_where, ) -from math_spec.dimensions import dims_of, pulled_back_dims +from math_spec.dimensions import dims_of, join_dims from math_spec.errors import DimensionError, LanguageError, SchemaError, case_context, did_you_mean, prefixed from math_spec.exclusivity import overlapping from math_spec.expansion import expand, parse_and_expand @@ -68,12 +68,12 @@ CountComparison, DimensionComparison, DimensionPosition, - Direction, Divide, Dual, Expression, ExpressionComparison, - GroupSum, + Join, + JoinColumns, Mask, Multiply, Named, @@ -88,7 +88,6 @@ Power, Predicate, PredicateOperator, - Pullback, PulledBackPredicate, Region, RelationComparison, @@ -332,7 +331,7 @@ class Resolved: constraints: Each constraint, as a program declares it. objective: The objective, ``None`` where the file declares none. relations: Each relation's columns and key, as declared — the one - copy, which every :class:`~math_spec.program.Direction` and + copy, which every :class:`~math_spec.program.JoinColumns` and :class:`~math_spec.program.Partition` in the trees holds. assumptions: Each ``assumptions:`` entry's predicate and the mask it is checked under. @@ -810,19 +809,19 @@ def _built( dims: Mapping[str, str | None], amounts: Mapping[str, int | str | None], edge: _Edge | None, - read: Direction | Partition | None, + read: JoinColumns | Partition | None, ) -> Expression | None: """The node *operator* builds from its read arguments, or ``None`` with the refusal appended.""" if operator == 'sum': if read is not None: - assert isinstance(read, Direction), 'a sum reads its relation in a direction' - return GroupSum(operand, read) + assert isinstance(read, JoinColumns), 'a sum joins its relation' + return Sum(Join(operand, read), read.axes) if (over := dims.get('over')) is not None: return Sum(operand, (over,)) return self._bare_sum(operand) if operator == 'at': - assert isinstance(read, Direction), 'at reads its relation in a direction' - return Pullback(operand, read) + assert isinstance(read, JoinColumns), 'at joins its relation' + return Join(operand, read) assert read is None or isinstance(read, Partition), 'a translation reads its relation as a partition' along = dims['along'] assert along is not None @@ -994,15 +993,15 @@ def _relation_ref( key: str, roles: Mapping[str, ArithmeticNode], along: str | None, - ) -> Direction | Partition | None: - """An operator's ``by=`` as the direction or the partition the call reads its relation in. + ) -> JoinColumns | Partition | None: + """An operator's ``by=`` as the join or the partition the call reads its relation in. A relation carries its own dimensions, so the call names columns rather - than dims: ``over=`` the column consumed, ``into=`` the column - produced, every other key column joined on. A value column not named - is not read, and a bare relation's columns are all key. One call - addresses one table, so several columns of one table are a list and - several tables are not. *along* is the dimension a translation steps + than dims: ``over=`` the columns joined on and summed away, ``into=`` + the columns grouped by, every other key column joined on and kept. A + value column not named is not read, and a bare relation's columns are + all key. One call addresses one table, so several columns of one table + are a list and several tables are not. *along* is the dimension a translation steps along, already read, or ``None`` where it was refused. """ names = names_in(value) @@ -1032,7 +1031,7 @@ def _relation_ref( return self._partition(name, operator, along, named['within']) if not ({'over', 'into'} <= set(named)): return None # refused already, by the call shape or by the role that named no column - return self._direction(name, operator, named['over'], named['into']) + return self._join(name, operator, named['over'], named['into']) def _role_name(self, value: ArithmeticNode, operator: str, key: str) -> tuple[str, ...] | None: """``over=`` or ``into=`` as the column names it must be — one bare name, or a bracketed list of them.""" @@ -1043,22 +1042,22 @@ def _role_name(self, value: ArithmeticNode, operator: str, key: str) -> tuple[st ) return None - def _direction( + def _join( self, name: str, operator: str, from_roles: tuple[str, ...], into_roles: tuple[str, ...], - ) -> Direction | None: - """Which direction ``sum`` or ``at`` reads relation *name* in, between the columns the call named. + ) -> JoinColumns | None: + """How ``sum`` or ``at`` joins relation *name*, between the columns the call named. Both ends arrive written: the call shape refuses a call that leaves one unsaid, so that a relation may gain a value column without - changing what this call means. ``at`` needs the read single-valued - and ``sum`` needs it not: a sum that lands on the key has one term - per coordinate and adds up nothing, which is a read, so it is - refused toward ``at``. A read lands on key columns and nothing else, - because a column outside the key is one no coordinate of the read + changing what this call means. A lookup needs every group to be one + row and a sum needs it not: a sum whose grouped columns hold the whole + key has one row per group and adds up nothing, which is a lookup, so + it is refused toward ``at``. A lookup groups by key columns and nothing + else, because a column outside the key is one no row of the join fixes. """ ns, context = self.ns, self.context @@ -1087,29 +1086,29 @@ def _direction( if not forward and (outside := [r for r in into_roles if r not in shape.key]): self.errors.append( f"{context}: {call}: into={list(into_roles)} names {outside}, which the key of '{name}' does not " - f'hold. A read lands on the key it reads at, {list(shape.key)}, and a column outside that key ' - f'arrives as a dimension the read never fixes. Land on the key, or sum toward {outside}.' + f'hold. A lookup reads one row per key, {list(shape.key)}, and a column outside that key ' + f'arrives as a dimension no row of the join fixes. Group by the key, or sum toward {outside}.' ) return None - joined = tuple(r for r in shape.key if r not in from_roles and r not in into_roles) - single_valued = set(shape.key) <= {*into_roles, *joined} - direction = Direction(name, shape, from_roles, into_roles, joined) - if not forward and not single_valued: + kept = tuple(r for r in shape.key if r not in from_roles and r not in into_roles) + join = JoinColumns(name, shape, (*from_roles, *kept), (*into_roles, *kept)) + one_row_per_group = set(shape.key) <= set(join.grouped) + if not forward and not one_row_per_group: self.errors.append( - f"{context}: {call}: at reads one value per coordinate, and '{name}' is not single-valued in " - f'{list(from_roles)} at the columns the call lands on ({[*into_roles, *joined]}) — its key is ' - f'{list(shape.key)}. Key the table by the columns the call lands on, or read the other way.' + f"{context}: {call}: at reads one row per group, and grouping '{name}' by {list(join.grouped)} " + f'leaves several rows in a group — its key is {list(shape.key)}. Key the table by the columns ' + f'the call groups by, or sum instead.' ) return None - if forward and single_valued: + if forward and one_row_per_group: self.errors.append( - f'{context}: {call}: this sum lands on the key {list(shape.key)}, so each coordinate has one ' - f"term and nothing is added up — that is a read, which is at()'s. Write " - f'at(..., by={name}, over={list(from_roles)}, into={list(into_roles)}), or sum toward ' - f'a value column.' + f'{context}: {call}: the columns this sum groups by, {list(join.grouped)}, hold the whole key ' + f'{list(shape.key)}, so every group is one row and nothing is added up — that is a join with no ' + f"group-by, which is at()'s. Write at(..., by={name}, over={list(from_roles)}, " + f'into={list(into_roles)}), or sum toward a value column.' ) return None - return direction + return join def _known_roles(self, name: str, call: str, roles: tuple[str, ...], kwarg: str) -> bool: """Whether every role *kwarg* names is a column of relation *name*, each once; the refusal otherwise.""" @@ -1323,10 +1322,10 @@ def _pulled_back(self, node: UnresolvedPredicateCallNode, mask: Mask) -> Predica found = len(self.errors) roles = {key: node.kwargs[key] for key in ('over', 'into')} by = self._relation_ref(node.kwargs['by'], 'at', 'by', roles, None) - if len(self.errors) > found or not isinstance(by, Direction): + if len(self.errors) > found or not isinstance(by, JoinColumns): return node try: - dims = pulled_back_dims(by, mask.dims, context, 'the predicate') + dims = join_dims(by, mask.dims, context, 'the predicate') except DimensionError as refusal: self.errors.append(str(refusal)) return node diff --git a/src/math_spec/separability.py b/src/math_spec/separability.py index 2c559b28..4534241e 100644 --- a/src/math_spec/separability.py +++ b/src/math_spec/separability.py @@ -11,9 +11,8 @@ from math_spec.program import ( Cases, DimensionPosition, - GroupSum, + Join, Mask, - Pullback, Reach, Separability, Sum, @@ -98,25 +97,26 @@ def waits_on(dimension: str, label: str, name: str, kind: Literal['offset', 'par if isinstance(node, Cases): masks.extend(region.when for region in node.regions) elif isinstance(node, Sum): - if reductions_couple: - for dimension in node.over: + join = node.operand.columns if isinstance(node.operand, Join) and node.operand.columns.axes else None + grouped = dict(zip(join.axes, join.dropped_dims, strict=True)) if join is not None else {} + for name in node.over: + if join is not None and name in grouped: report( 'coupled', - dimension, + grouped[name], label, - f'sums over {dimension} — a rolling sum_back(window=n) windows, a total over the horizon does not', + f'groups {grouped[name]} into {", ".join(join.added_dims)} — window that dimension instead, or cut only at the group edges', ) - elif isinstance(node, GroupSum): - for dimension in node.direction.consumed_dims: - report( - 'coupled', - dimension, - label, - f'groups {dimension} into {", ".join(node.direction.produced_dims)} — window that dimension instead, or cut only at the group edges', - ) - elif isinstance(node, Pullback): - for dimension in node.direction.consumed_dims: - waits_on(dimension, label, node.direction.name, 'coordinate') + elif reductions_couple: + report( + 'coupled', + name, + label, + f'sums over {name} — a rolling sum_back(window=n) windows, a total over the horizon does not', + ) + elif isinstance(node, Join) and not node.columns.axes: + for dimension in node.columns.dropped_dims: + waits_on(dimension, label, node.columns.name, 'coordinate') elif isinstance(node, (Translate, WindowSum)): dimension = node.along if node.wrap: diff --git a/src/math_spec/typesetting/walk.py b/src/math_spec/typesetting/walk.py index 7d018928..b3727557 100644 --- a/src/math_spec/typesetting/walk.py +++ b/src/math_spec/typesetting/walk.py @@ -26,12 +26,12 @@ CountComparison, DimensionComparison, DimensionPosition, - Direction, Divide, Dual, Expression, ExpressionComparison, - GroupSum, + Join, + JoinColumns, Mask, Multiply, Named, @@ -45,7 +45,6 @@ Power, Predicate, PredicateOperator, - Pullback, PulledBackPredicate, RelationComparison, RelationDefined, @@ -159,8 +158,8 @@ class _Context: walk: Walk offsets: dict[str, tuple[_Step, ...]] = field(default_factory=dict) - #: dim -> the rendered subscript that replaces its index, as ``at`` re-indexes a leaf. - pullbacks: dict[str, str] = field(default_factory=dict) + #: dim -> the rendered subscript that replaces its index, as ``at`` looks a leaf up through a relation. + lookups: dict[str, str] = field(default_factory=dict) #: Every dimension whose index is in use here — the frame, then one entry #: per reduction entered — so a reduction over one takes a fresh dummy. bound: tuple[str, ...] = () @@ -169,10 +168,10 @@ def translated(self, dim: str, step: _Step) -> _Context: steps = self.offsets.get(dim, ()) merged = steps[-1].merged(step) if steps else None steps = (*steps[:-1], merged) if merged is not None else (*steps, step) - return _Context(self.walk, {**self.offsets, dim: steps}, self.pullbacks, self.bound) + return _Context(self.walk, {**self.offsets, dim: steps}, self.lookups, self.bound) - def pulled_back(self, dim: str, rendered: str) -> _Context: - return _Context(self.walk, self.offsets, {**self.pullbacks, dim: rendered}, self.bound) + def looked_up(self, dim: str, rendered: str) -> _Context: + return _Context(self.walk, self.offsets, {**self.lookups, dim: rendered}, self.bound) def reducing(self, dim: str) -> tuple[str, _Context]: """A dummy index for a reduction over *dim*, and the context its body reads under. @@ -183,18 +182,18 @@ def reducing(self, dim: str) -> tuple[str, _Context]: """ primes = "'" * self.bound.count(dim) dummy = f'{self.walk.symbols.index[dim]}{primes}' - body = _Context(self.walk, self.offsets, {**self.pullbacks, dim: dummy}, (*self.bound, dim)) + body = _Context(self.walk, self.offsets, {**self.lookups, dim: dummy}, (*self.bound, dim)) return dummy, body def subscript(self, dim: str) -> str: - """The index for *dim* here: its pullback if it has one, then every translation. + """The index for *dim* here: its lookup if it has one, then every translation. - A pullback is a base like any other rather than a stopping point. + A lookup is a base like any other rather than a stopping point. ``at`` and ``shift`` both re-index the leaf and the leaf has one subscript, so a reading that showed only whichever ran last dropped the other operator out of the equation. """ - text = self.pullbacks.get(dim, self.walk.symbols.index[dim]) + text = self.lookups.get(dim, self.walk.symbols.index[dim]) translated = False for step in self.offsets.get(dim, ()): if step.by == 0: @@ -367,11 +366,8 @@ def _arithmetic(self, node: Expression, ctx: _Context) -> tuple[str, int]: if isinstance(node, Sum): return self._sum(node, ctx) - if isinstance(node, GroupSum): - return self._group_sum(node, ctx) - - if isinstance(node, Pullback): - return self._pullback(node, ctx) + if isinstance(node, Join): + return self._join(node, ctx) if isinstance(node, Translate): return self._translate(node, ctx) @@ -427,7 +423,9 @@ def _binary(self, node: Add | Multiply | Divide | Power, ctx: _Context) -> tuple return self.format.joined([left, right], self._op(names[op])), precedence def _sum(self, node: Sum, ctx: _Context) -> tuple[str, int]: - """A reduction over named dims: one dummy index per dim, in declaration order.""" + """A reduction over named dims: one dummy index per dim, in declaration order, or the group-by over a join.""" + if isinstance(node.operand, Join) and node.operand.columns.axes: + return self._grouped_sum(node.operand, ctx) memberships = [] inner = ctx for d in self._sorted(frozenset(node.over)): @@ -436,23 +434,23 @@ def _sum(self, node: Sum, ctx: _Context) -> tuple[str, int]: domain = self.format.joined(memberships, '') return self.format.summation(domain, self._reduction_body(node.operand, inner)), _PRECEDENCE['+'] - def _group_sum(self, node: GroupSum, ctx: _Context) -> tuple[str, int]: - """A sum through a relation: a dummy per consumed dim, and the row it joins on as the domain's condition.""" - direction = node.direction + def _grouped_sum(self, join: Join, ctx: _Context) -> tuple[str, int]: + """A sum over the axes a join opens: a dummy per dim it drops, and the row it joins on as the domain's condition.""" + columns = join.columns dummies: dict[str, str] = {} inner = ctx - for d in direction.consumed_dims: + for d in columns.dropped_dims: dummies[d], inner = inner.reducing(d) - conditions = list(self._grouping(direction, dummies, ctx)) + conditions = list(self._grouping(columns, dummies, ctx)) domain = ( - f'{self.format.joined([self._membership(d, dummies[d]) for d in direction.consumed_dims], "")} ' + f'{self.format.joined([self._membership(d, dummies[d]) for d in columns.dropped_dims], "")} ' f'{self._op("such_that")} {self.format.joined(conditions, self._op("and"))}' ) - return self.format.summation(domain, self._reduction_body(node.operand, inner)), _PRECEDENCE['+'] + return self.format.summation(domain, self._reduction_body(join.operand, inner)), _PRECEDENCE['+'] - def _pullback(self, node: Pullback, ctx: _Context) -> tuple[str, int]: - """``at`` emits no operator of its own: it re-indexes the operand, so the read shows at the leaves.""" - return self._arithmetic(node.operand, self._pulled_back(node.direction, ctx)) + def _join(self, node: Join, ctx: _Context) -> tuple[str, int]: + """``at`` emits no operator of its own: it re-indexes the operand, so the lookup shows at the leaves.""" + return self._arithmetic(node.operand, self._looked_up(node.columns, ctx)) def _translate(self, node: Translate, ctx: _Context) -> tuple[str, int]: """``shift`` emits no operator of its own: it re-indexes the operand, so the translation shows at the leaves. @@ -481,30 +479,30 @@ def _window_sum(self, node: WindowSum, ctx: _Context) -> tuple[str, int]: body = self._reduction_body(node.operand, inner) return self.format.summation(domain, body), _PRECEDENCE['+'] - def _pulled_back(self, direction: Direction, ctx: _Context) -> _Context: - """*ctx* with each dimension *direction* consumes read at the relation, as ``at`` re-indexes a leaf.""" - at = {r: ctx.subscript(direction.dim(r)) for r in (*direction.produced, *direction.joined)} - for read in direction.consumed: - ctx = ctx.pulled_back(direction.dim(read), self._relation_read(direction.name, at, read)) + def _looked_up(self, columns: JoinColumns, ctx: _Context) -> _Context: + """*ctx* with each dimension *columns* drops read at the relation, as ``at`` re-indexes a leaf.""" + at = {r: ctx.subscript(columns.dim(r)) for r in columns.grouped} + for read in columns.dropped: + ctx = ctx.looked_up(columns.dim(read), self._relation_read(columns.name, at, read)) return ctx - def _grouping(self, direction: Direction, dummies: Mapping[str, str], ctx: _Context) -> list[str]: - """The conditions a grouped sum's domain carries for one direction: what it fixes of the row it joins on. + def _grouping(self, join: JoinColumns, dummies: Mapping[str, str], ctx: _Context) -> list[str]: + """The conditions a grouped sum's domain carries for one join: what it fixes of the row it joins on. - A direction fixes its relation's key either way, so each value column - it fixes — consumed or produced, one lookup at the key either way — is + A join fixes its relation's key either way, so each value column it + names — joined on or grouped by, one read at the key either way — is that column read there. One that fixes none of them asks only that the row is there, because a value column it does not touch is not read, and a bare relation has no value column to read at all. """ at = { - **{r: dummies[direction.dim(r)] for r in direction.consumed}, - **{r: ctx.subscript(direction.dim(r)) for r in (*direction.joined, *direction.produced)}, + **{r: dummies[join.dim(r)] for r in join.dropped}, + **{r: ctx.subscript(join.dim(r)) for r in join.grouped}, } - fixed = [r for r in self.schema.relations[direction.name].value_roles if r in at] + fixed = [r for r in self.schema.relations[join.name].value_roles if r in at] if not fixed: - return [self._relation_row(direction.name, at)] - return [f'{self._relation_read(direction.name, at, r)} {self._op("equal")} {at[r]}' for r in fixed] + return [self._relation_row(join.name, at)] + return [f'{self._relation_read(join.name, at, r)} {self._op("equal")} {at[r]}' for r in fixed] def _group(self, partition: Partition | None) -> str: """A ``by=`` as the superscript its translation operator carries. @@ -516,7 +514,7 @@ def _group(self, partition: Partition | None) -> str: if partition is None: return '' at = {r: self.symbols.index[partition.dim(r)] for r in (partition.along, *partition.joined)} - return self._tuple([self._relation_read(partition.name, at, r) for r in partition.group]) + return self._tuple([self._relation_read(partition.name, at, r) for r in partition.grouped]) def _width(self, width: int | str) -> str: """``sum_back``'s ``window=``: a number, or a parameter's own symbol. @@ -576,7 +574,7 @@ def _where(self, node: Predicate, ctx: _Context) -> tuple[str, int]: return self._where(node.operand.root, moved) if isinstance(node, PulledBackPredicate): - return self._where(node.operand.root, self._pulled_back(node.direction, ctx)) + return self._where(node.operand.root, self._looked_up(node.columns, ctx)) if isinstance(node, RelationDefined): return self._relation_row(node.name, self._frame_key(node.name, ctx)), comparison @@ -617,7 +615,7 @@ def sides(self, node: AlignedComparison, ctx: _Context) -> tuple[str, str]: grouping = ( None if node.partition is None - else self._tuple([self._value_read(node.partition.name, c, ctx) for c in node.partition.group]) + else self._tuple([self._value_read(node.partition.name, c, ctx) for c in node.partition.grouped]) ) left = self._position(ctx.subscript(node.name), grouping) right = self._ordinal(node.name, node.position, grouping) diff --git a/tests/fixtures/every_program_node.yaml b/tests/fixtures/every_program_node.yaml index a371b705..ab3a5994 100644 --- a/tests/fixtures/every_program_node.yaml +++ b/tests/fixtures/every_program_node.yaml @@ -38,7 +38,7 @@ constraints: grouped: dims: [t, zone] expression: "sum(p, by=zone_of, over=g, into=zone) - q <= load" - pulled_back: + looked_up: dims: [t, g] expression: "p - at(q, by=zone_of, over=zone, into=g) <= 0" translated: diff --git a/tests/test_advice.py b/tests/test_advice.py index 37232fd1..49019c22 100644 --- a/tests/test_advice.py +++ b/tests/test_advice.py @@ -52,7 +52,7 @@ def test_a_dimension_nothing_reaches_is_named(): ) def test_a_dimension_something_reaches_is_in_use(patch): assert not advice(override(TARGET_ONLY, **patch)), ( - 'a dimension a relation targets, a declaration indexes or a grouping lands on is in use' + 'a dimension a relation targets, a declaration indexes or a grouping groups by is in use' ) diff --git a/tests/test_degree.py b/tests/test_degree.py index 3dfb61d2..b7b7ce61 100644 --- a/tests/test_degree.py +++ b/tests/test_degree.py @@ -72,6 +72,7 @@ def test_the_affine_ceiling_refuses_and_names_the_rewrite(text, fragment): pytest.param('(p + q) * c * p', id='a-sum-against-one-term'), pytest.param('p * q / c', id='a-quadratic-over-a-parameter'), pytest.param('p * r * c', id='a-broadcast-product-of-disjoint-dims'), + pytest.param('at(r, by=lk, over=h, into=g) * at(r, by=lk, over=h, into=g)', id='two-lookups-are-one-term-each'), ], ) def test_the_objective_takes_degree_two(text): @@ -84,6 +85,9 @@ def test_the_objective_takes_degree_two(text): pytest.param('p * q * p', 'this product is degree 3', id='a-cubic'), pytest.param('(p * q) * (p * q)', 'this product is degree 4', id='a-quartic'), pytest.param('sum(p, over=g) * sum(q, over=g)', 'outer product', id='two-reductions'), + pytest.param( + 'sum(p, by=lk, over=g, into=h) * sum(q, over=g)', 'outer product', id='a-grouped-sum-is-a-reduction' + ), pytest.param('(p + q) * (p + q)', 'outer product', id='two-sums-of-variables'), pytest.param('sum_back(p, along=g, window=1) * (p - q)', 'outer product', id='a-window-against-a-difference'), ], diff --git a/tests/test_dimensions.py b/tests/test_dimensions.py index b286b362..2f56c298 100644 --- a/tests/test_dimensions.py +++ b/tests/test_dimensions.py @@ -12,7 +12,7 @@ from math_spec.dimensions import DimensionError, _check_where_dims, dims_of from math_spec.errors import SchemaError -from math_spec.program import Mask, RelationPairComparison +from math_spec.program import Join, Mask, RelationPairComparison, Sum from math_spec.resolution import Namespace from math_spec.validation import to_spec from tests.fixtures import expression_of, override, schema_of, where_of @@ -112,7 +112,7 @@ def namespace() -> Namespace: pytest.param( 'sum(p, by=gen_zone, over=generator, into=zone)', {'snapshot', 'zone'}, - id='a-two-key-relation-consumes-the-key-it-names-and-keeps-the-other', + id='a-two-key-relation-sums-away-the-key-it-names-and-keeps-the-other', ), pytest.param( 'sum(p, by=gen_zone, over=snapshot, into=zone)', @@ -122,7 +122,7 @@ def namespace() -> Namespace: pytest.param( 'at(zone_load, by=gen_zone, into=generator, over=zone)', {'snapshot', 'generator'}, - id='its-pullback-keeps-the-joined-key-too', + id='its-lookup-keeps-the-joined-key-too', ), pytest.param( "shift(p, along=generator, offset=1, edge='wrap', by=gen_zone, within=zone)", @@ -142,7 +142,7 @@ def namespace() -> Namespace: pytest.param( "shift(p, along=generator, offset=1, edge='wrap', by=pair, within=[b0, b1])", {'snapshot', 'generator'}, - id='a-partition-grouped-by-two-columns-over-one-dimension-lands-nothing', + id='a-partition-grouped-by-two-columns-over-one-dimension-adds-nothing', ), pytest.param( 'sum(p, by=gen_bus, over=generator, into=bus)', @@ -162,7 +162,7 @@ def namespace() -> Namespace: pytest.param( 'sum(p, by=gen_zone, over=[generator, snapshot], into=zone)', {'zone'}, - id='a-from-list-consumes-two-key-columns-at-once', + id='an-over-list-joins-on-two-key-columns-at-once', ), pytest.param( 'sum(p, by=gen_bz, into=bus, over=generator)', @@ -180,7 +180,7 @@ def namespace() -> Namespace: id='a-map-into-its-own-dimension-keeps-the-frame', ), pytest.param( - 'at(p, by=rep_of, over=rep, into=snapshot)', {'snapshot', 'generator'}, id='and-so-does-its-pullback' + 'at(p, by=rep_of, over=rep, into=snapshot)', {'snapshot', 'generator'}, id='and-so-does-its-lookup' ), pytest.param( "shift(p, along=snapshot, offset=1, edge='wrap', by=rep_of, within=rep)", @@ -204,7 +204,7 @@ def _dims_with(expr: str, **overrides) -> frozenset[str]: pytest.param( 'sum(p, by=connection, over=generator, into=bus)', {'snapshot', 'bus'}, - id='a-sum-through-a-bare-relation-lands-on-a-key-column', + id='a-sum-through-a-bare-relation-groups-by-a-key-column', ), pytest.param( 'sum(load, by=connection, over=bus, into=generator)', @@ -214,22 +214,39 @@ def _dims_with(expr: str, **overrides) -> frozenset[str]: ], ) def test_a_bare_relation_is_summed_between_its_key_columns(expr, expected): - """A bare relation holds no value column, so the column a sum lands on is a key column.""" + """A bare relation holds no value column, so the column a sum groups by is a key column.""" assert _dims_with(expr, **{'relations.connection': {'key': ['generator', 'bus']}}) == expected -def test_a_read_carries_the_whole_key_and_what_the_operand_brings_beside_it(): - """A read lands on the key however the call splits it, and a dim the operand carries and the read does not consume rides along. +def test_a_lookup_carries_the_whole_key_and_what_the_operand_brings_beside_it(): + """A lookup groups by the key however the call splits it, and a dim the operand carries and the lookup does not join on rides along. - `gen_bz` is keyed by `generator` alone, so the key is produced whole; the - operand's `snapshot` is neither consumed nor part of the key, and the + `gen_bz` is keyed by `generator` alone, so the whole key is grouped by; the + operand's `snapshot` is neither joined on nor part of the key, and the result keeps it. """ assert _dims('at(zone_load, by=gen_bz, over=zone, into=generator)') == {'generator', 'snapshot'} -def test_a_sum_consumes_a_key_column_and_a_value_column_together(): - """A sum's consumed end is not one kind of column: it needs one key column, and may name a value column beside it.""" +def test_a_join_opens_an_axis_for_the_column_it_drops_and_the_sum_over_it_closes_it(): + """A map into its own dimension drops and adds one dimension, so the join names the dropped column for the relation. + + Named for its dimension, the column the join drops and the column it + groups by are one name, and the sum over the join takes away the dim the + row keeps. + """ + s = _schema() + node = expression_of('sum(p, by=rep_of, over=snapshot, into=rep)', Namespace(s), 't') + assert isinstance(node, Sum) and isinstance(node.operand, Join) + assert node.over == ('rep_of.snapshot',), 'the sum stands over the axis the join opens' + assert dims_of(node.operand, s, 't') == {'generator', 'snapshot', 'rep_of.snapshot'}, ( + 'the join keeps the dropped column beside the dimension it groups by' + ) + assert dims_of(node, s, 't') == {'generator', 'snapshot'}, 'and the sum over it leaves the frame the row keeps' + + +def test_a_sum_joins_on_a_key_column_and_a_value_column_together(): + """The columns a sum joins on and sums away are not one kind: it needs one key column, and may name a value column beside it.""" assert _dims('sum(p * load, by=gen_bz, over=[generator, bus], into=zone)') == {'snapshot', 'zone'} @@ -253,7 +270,7 @@ def test_a_bare_name_reaches_the_variable_a_dual_the_same_named_constraint(): 'sum(p, over=bus)', DimensionError, r'sum\(over=bus\) but the expression has dims', - id='sum-consuming-an-absent-dim-is-an-error-not-a-noop', + id='sum-over-an-absent-dim-is-an-error-not-a-noop', ), pytest.param( 'sum(sum(p))', @@ -276,7 +293,7 @@ def test_a_bare_name_reaches_the_variable_a_dual_the_same_named_constraint(): pytest.param( 'sum(load, by=gen_bus, over=generator, into=bus)', DimensionError, - r"sum\(by=gen_bus\) consumes \['generator'\], the dims it reads from", + r"sum\(by=gen_bus\) joins on \['generator'\] to sum it away", id='sum-requires-the-grouped-dim', ), pytest.param( @@ -337,7 +354,7 @@ def test_a_bare_name_reaches_the_variable_a_dual_the_same_named_constraint(): 'at(zone_cap, by=gen_zone, into=generator, over=zone)', DimensionError, r"at\(by=gen_zone\) joins on \['snapshot'\]", - id='a-pullback-needs-the-keys-it-joins-on', + id='a-lookup-needs-the-keys-it-joins-on', ), pytest.param( "shift(cost, along=generator, offset=1, edge='wrap', by=gen_zone, within=zone)", @@ -364,24 +381,24 @@ def test_an_ill_dimensioned_expression_is_rejected(expr, error, match): pytest.param( 'sum(p, by=diag, over=k, into=z)', {'key': {'k': 'generator', 'j': 'generator', 'z': 'zone'}}, - id='a-sum-consuming-a-column-over-the-dimension-it-joins-on', + id='a-sum-summing-away-a-column-over-a-dimension-it-also-joins-on', ), pytest.param( 'at(load, by=diag, over=rep, into=generator)', {'key': ['snapshot', 'generator'], 'values': {'rep': 'snapshot'}}, - id='a-read-consuming-a-column-over-the-dimension-it-joins-on', + id='a-lookup-reading-a-column-over-a-dimension-it-also-joins-on', ), ], ) -def test_a_joined_column_is_not_also_consumed(expr, diag): - """The operand carries one coordinate per dimension, so a column consumed and a column joined on cannot share one. - - The `at` case passed: the check asked whether a joined dimension was - *produced*, which the landing check already refuses, and not whether it - was consumed. `at(load, by=diag, over=rep, into=generator)` then read - `rep` at the operand's snapshot and joined on the key's snapshot at the - same coordinate, and landed on `[bus, generator]` with the joined - dimension gone. +def test_two_joined_columns_do_not_share_a_dimension(expr, diag): + """The operand carries one coordinate per dimension, so the `over=` column and an unnamed key column cannot share one. + + The `at` case passed: the check asked whether an unnamed key dimension + was also grouped by, which the grouping check already refuses, and not + whether it was also the one read. `at(load, by=diag, over=rep, into=generator)` + then read `rep` at the operand's snapshot and joined on the key's snapshot + at the same coordinate, and came out over `[bus, generator]` with the + joined dimension gone. """ with pytest.raises(DimensionError, match=r"joins 'diag' on \[.*\] through more than one column"): _dims_with(expr, **{'relations.diag': diag}) diff --git a/tests/test_lowering.py b/tests/test_lowering.py index cd2d1258..17d5e81e 100644 --- a/tests/test_lowering.py +++ b/tests/test_lowering.py @@ -31,13 +31,13 @@ CountComparison, DimensionComparison, DimensionDeclaration, - Direction, Divide, Dual, Expression, ExpressionComparison, Footprint, - GroupSum, + Join, + JoinColumns, Mask, Multiply, Named, @@ -50,7 +50,6 @@ Partition, Power, Program, - Pullback, PulledBackPredicate, Region, RelationDeclaration, @@ -86,15 +85,15 @@ 'constraints': {'c': {'dims': [], 'expression': 'sum(p, over=g) >= 1'}}, } -#: `lk` as `sum` reads it: key consumed, value produced, nothing joined. +#: `lk` as `sum` joins it: joined on the key, grouped by the value, no key column left unnamed. LK = RelationDeclaration((('g', 'g'), ('h', 'h')), ('g',)) LK2 = RelationDeclaration((('g', 'g'), ('z', 'z')), ('g',)) -LK_DIRECTION = Direction('lk', LK, ('g',), ('h',), ()) +LK_JOIN = JoinColumns('lk', LK, ('g',), ('h',)) AT_BUS = RelationDeclaration((('g', 'g'), ('bus', 'bus')), ('g',)) #: `fixtures.SMALL_MODEL` plus a second relation and a per-entity #: offset. Which node a construct becomes is mostly a claim about the dim it -#: consumes and the dim it lands on, and stating that needs a third dimension +#: joins on and the dim it groups by, and stating that needs a third dimension #: and two relations over one of them. SHAPES_MODEL = override( SMALL_MODEL, @@ -424,7 +423,7 @@ def test_a_comparison_of_expressions_lowers_to_program_expressions_on_both_sides ) mask = program.constraints['w'].where assert mask is not None and isinstance(mask.root, ExpressionComparison) - assert isinstance(mask.root.right, Add) and isinstance(mask.root.right.left, Pullback) + assert isinstance(mask.root.right, Add) and isinstance(mask.root.right.left, Join) assert mask.names_read == frozenset({'c', 'zc', 'lk2'}), ( 'the relation a pullback and a partition read through is data the consumer binds too' ) @@ -576,17 +575,17 @@ def test_a_power_resolves_to_a_node_of_its_own(dispatch_schema): @pytest.mark.parametrize( ('expression', 'expected'), [ - pytest.param('sum(q)', Sum(Variable('q'), ('g', 'h')), id='a-bare-sum-consumes-every-dim-the-operand-carries'), - pytest.param('sum(q, over=h)', Sum(Variable('q'), ('h',)), id='an-over-consumes-the-dim-it-names'), + pytest.param('sum(q)', Sum(Variable('q'), ('g', 'h')), id='a-bare-sum-sums-away-every-dim-the-operand-carries'), + pytest.param('sum(q, over=h)', Sum(Variable('q'), ('h',)), id='an-over-sums-away-the-dim-it-names'), pytest.param( 'sum(p, by=lk, over=g, into=h)', - GroupSum(Variable('p'), direction=LK_DIRECTION), - id='a-grouped-sum-names-the-dim-it-consumes-and-the-one-it-lands-on', + Sum(Join(Variable('p'), LK_JOIN), ('lk.g',)), + id='a-grouped-sum-is-a-sum-over-the-axis-its-join-opens', ), pytest.param( 'at(r, by=lk, over=h, into=g)', - Pullback(Variable('r'), direction=Direction('lk', LK, ('h',), ('g',), ())), - id='a-pullback-reads-the-same-table-back', + Join(Variable('r'), JoinColumns('lk', LK, ('h',), ('g',))), + id='an-at-is-the-same-join-the-other-way-with-no-sum-over-it', ), pytest.param( "shift(p, along=g, offset=1, edge='wrap')", @@ -670,7 +669,7 @@ def test_a_partition_keeps_its_group_when_the_relation_gains_a_value_column(): }, } ) - grouping[str(values)] = _partition_of(program.constraints['k']).group + grouping[str(values)] = _partition_of(program.constraints['k']).grouped assert grouping == {'day': ('day',), "['day', 'week']": ('day',)}, ( 'the group is the columns the call named, on both calendars' ) @@ -683,8 +682,8 @@ def _partition_of(row): return partition -def test_a_relation_lowers_with_the_direction_each_call_names(): - """Every node reading a relation carries its columns, its key and the direction, so a consumer joins on the right columns.""" +def test_a_relation_lowers_with_the_join_each_call_names(): + """Every node reading a relation carries its columns, its key and the join, so a consumer joins on the right columns.""" program = to_program( { 'dimensions': {'snapshot': {'dtype': 'int'}, 'generator': {}, 'zone': {}}, @@ -718,31 +717,35 @@ def test_a_relation_lowers_with_the_direction_each_call_names(): declared = RelationDeclaration(columns, ('generator', 'snapshot')) assert program.relations == {'zone_of': declared}, 'the relation sits once in the program, under its name' zonal = program.constraints['zonal'].lhs - assert zonal == GroupSum( - Variable('p'), direction=Direction('zone_of', declared, ('generator',), ('zone',), ('snapshot',)) - ), 'a grouped sum names the column it consumes, the one it produces and the one it joins on' - assert isinstance(zonal, GroupSum) - assert (zonal.direction.consumed_dims, zonal.direction.produced_dims, zonal.direction.joined_dims) == ( + columns = JoinColumns('zone_of', declared, ('generator', 'snapshot'), ('zone', 'snapshot')) + assert zonal == Sum(Join(Variable('p'), columns), ('zone_of.generator',)), ( + 'a grouped sum is a sum over a join: the join names the over= column and the unnamed key column as joined ' + 'on, the into= column and that key column as grouped by, and the sum stands over the axis the join opens ' + 'for the column it drops' + ) + assert isinstance(zonal, Sum) and isinstance(zonal.operand, Join) + assert (zonal.operand.columns.dropped_dims, zonal.operand.columns.added_dims, zonal.operand.columns.kept) == ( ('generator',), ('zone',), ('snapshot',), - ), 'the dims a consumer reads are read off the direction' - assert zonal.direction.relation is program.relations['zone_of'], ( - 'the direction holds the one declaration the program holds, not an equal copy built again' + ), 'the dims a consumer reads are read off the join: dropped, added, and the key columns kept' + assert zonal.operand.columns.relation is program.relations['zone_of'], ( + 'the join holds the one declaration the program holds, not an equal copy built again' ) - assert program.constraints['history'].lhs == GroupSum( - Variable('p'), direction=Direction('zone_of', declared, ('snapshot',), ('zone',), ('generator',)) - ), 'the same table read from its other key column' + assert program.constraints['history'].lhs == Sum( + Join(Variable('p'), JoinColumns('zone_of', declared, ('snapshot', 'generator'), ('zone', 'generator'))), + ('zone_of.snapshot',), + ), 'the same table joined on its other key column' priced = program.constraints['priced'].rhs - assert priced == Pullback( - Parameter('price'), direction=Direction('zone_of', declared, ('zone',), ('generator',), ('snapshot',)) - ), 'and its adjoint consumes the value column and produces the key column' - assert isinstance(priced, Pullback) - assert (priced.direction.consumed_dims, priced.direction.produced_dims, priced.direction.joined_dims) == ( + assert priced == Join( + Parameter('price'), JoinColumns('zone_of', declared, ('zone', 'snapshot'), ('generator', 'snapshot')) + ), 'and an at is the bare join, on the value column, grouped by the key columns' + assert isinstance(priced, Join) + assert (priced.columns.dropped_dims, priced.columns.added_dims, priced.columns.kept) == ( ('zone',), ('generator',), ('snapshot',), - ), 'an at consumes the coarse dims, produces the fine, and joins on the rest of the key' + ), 'an at joins on the coarse dims, groups by the fine, and keeps the rest of the key' p_where = program.variables['p'].where assert p_where is not None assert [(type(a).__name__, a.dims) for a in p_where.atoms] == [ @@ -761,14 +764,14 @@ def test_a_binary_variable_lowers_to_a_binary_domain(): assert program.variables['dispatch'].domain == 'binary' -def test_a_divisor_under_a_pullback_is_still_named(): +def test_a_divisor_under_a_join_is_still_named(): """`children` has to descend through every node, or a refusal loses its name.""" quotient = Divide(Variable('x'), Parameter('rate')) component_of = RelationDeclaration((('flow', 'flow'), ('component', 'component')), ('flow',)) - pulled = Pullback(quotient, direction=Direction('component_of', component_of, ('component',), ('flow',), ())) + looked_up = Join(quotient, JoinColumns('component_of', component_of, ('component',), ('flow',))) - assert divisor_parameters(pulled) == frozenset({'rate'}), 'the walk descends through `Pullback`' - assert divisor_parameters(Sum(pulled, ('flow',))) == frozenset({'rate'}), 'and through a `Sum` over it' + assert divisor_parameters(looked_up) == frozenset({'rate'}), 'the walk descends through `Join`' + assert divisor_parameters(Sum(looked_up, ('flow',))) == frozenset({'rate'}), 'and through a `Sum` over it' def test_a_divisor_under_a_power_is_still_named(): @@ -840,8 +843,8 @@ def test_walk_is_the_node_column_of_walk_regions(): Power(Parameter('c'), Constant(2.0)): 'one-to-one', Divide(Variable('p'), Parameter('c')): 'one-to-one', Sum(Variable('p'), ('g',)): 'many-to-one', - GroupSum(Variable('p'), direction=Direction('at_bus', AT_BUS, ('g',), ('bus',), ())): 'many-to-one', - Pullback(Variable('p'), direction=Direction('at_bus', AT_BUS, ('bus',), ('g',), ())): 'one-to-one', + Sum(Join(Variable('p'), JoinColumns('at_bus', AT_BUS, ('g',), ('bus',))), ('at_bus.g',)): 'many-to-one', + Join(Variable('p'), JoinColumns('at_bus', AT_BUS, ('bus',), ('g',))): 'one-to-one', Translate(Variable('p'), 't', offset=1, wrap=False, fill=0.0): 'one-to-one', WindowSum(Variable('p'), 't', width=2, wrap=False): 'one-to-many', Cases((Region(Mask(ParameterDefined('c', ('g',))), Variable('p')),)): 'one-to-one', diff --git a/tests/test_piecewise.py b/tests/test_piecewise.py index 9166e9b2..4779ed56 100644 --- a/tests/test_piecewise.py +++ b/tests/test_piecewise.py @@ -420,7 +420,7 @@ def test_a_link_reading_a_dual_entry_is_refused(): @pytest.mark.parametrize( ('activity', 'match'), [ - pytest.param('at(u_unit, by=unit_of)', 'is not a declared variable', id='a-pullback-through-a-relation'), + pytest.param('at(u_unit, by=unit_of)', 'is not a declared variable', id='a-lookup-through-a-relation'), pytest.param('shift(u, along=snapshot, offset=1)', 'is not a declared variable', id='a-shifted-gate'), pytest.param('u * 2', 'is not a declared variable', id='an-arithmetic-gate'), ], diff --git a/tests/test_separability.py b/tests/test_separability.py index 02041975..f3a649e2 100644 --- a/tests/test_separability.py +++ b/tests/test_separability.py @@ -159,6 +159,24 @@ def test_a_read_through_a_relation_is_undecided_on_the_axis_it_reads(): ) +def test_a_sum_over_a_lookup_is_a_sum_and_a_lookup_not_a_grouping(): + """A plain `sum` over an `at` lowered to the `Sum` over a `Join` that a grouped sum lowered to. + + Separability read the shape, so it reported a grouping of `u` into `u` and + lost the read of `zone` the lookup waits on. Which call a join is comes + from its columns, not from the node above it. + """ + variables = {**BASE['variables'], 'q': {'dims': ['h', 'zone'], 'bounds': {'lower': 0}}} + rows = _rows('sum(at(q, by=zone_of, over=zone, into=u), over=u) <= budget', dims=['h']) + program = ms.to_program({**BASE, 'variables': variables, **rows}) + assert list(program.separability['u'].coupled.values()) == [ + 'sums over u — a rolling sum_back(window=n) windows, a total over the horizon does not' + ], 'the sum over u is a plain sum, reported as one' + assert program.separability['zone'].undecided == (Reach("constraint 'k'", 'zone_of', 'coordinate'),), ( + 'the lookup under it still reads zone at a coordinate the relation chooses' + ) + + def test_a_coupling_names_the_change_that_would_lift_it(): coupled = _verdict(**_rows('sum(p, over=h) <= budget', dims=['u'])).coupled["constraint 'k'"] assert 'sum_back(window=n)' in coupled, 'a horizon total becomes a rolling one' @@ -212,7 +230,7 @@ def test_the_lookahead_is_the_widest_reach_of_any_block(): assert verdict.ahead == 5, 'one window must see past its last row as far as any block reads' -def test_a_grouping_that_consumes_the_axis_couples_it(): +def test_a_grouping_that_sums_the_axis_away_couples_it(): program = ms.to_program( { **BASE, @@ -220,7 +238,11 @@ def test_a_grouping_that_consumes_the_axis_couples_it(): } ) verdict = program.separability['u'] - assert not verdict.windowable, 'the grouping consumes u, so a window of u is a different sum' + assert not verdict.windowable, 'the grouping sums u away, so a window of u is a different sum' + assert verdict.coupled == { + "constraint 'z'": 'groups u into zone — window that dimension instead, or cut only at the group edges' + }, 'the sum over the join is the grouping, reported once' + assert not verdict.undecided, 'the join under the sum is not also a lookup waiting on the relation' def test_every_declared_axis_has_a_verdict_and_nothing_else_does(): @@ -247,7 +269,7 @@ def test_a_reduction_over_several_axes_couples_every_one_of_them(): so the verdict for each of them has to say so — a walk that read only the first would call the rest windowable.""" program = ms.to_program({**BASE, 'constraints': {'all': {'dims': [], 'expression': 'sum(p) <= budget'}}}) - assert not program.separability['h'].windowable, 'the reduction consumes h' + assert not program.separability['h'].windowable, 'the reduction sums h away' assert not program.separability['u'].windowable, 'and u, in the same node' diff --git a/tests/test_validation.py b/tests/test_validation.py index d9ab9993..3ef0d910 100644 --- a/tests/test_validation.py +++ b/tests/test_validation.py @@ -901,12 +901,12 @@ def test_a_shape_the_language_admits(self, where): ), pytest.param( 'at(flag, by=lk, over=h, into=g)', - ("at(by=lk) reads through ['h'], which the predicate does not carry",), + ("at(by=lk) joins on ['h'], which the predicate does not carry",), id='a-read-through-a-dim-the-predicate-lacks', ), pytest.param( 'at(q, by=lk, over=h, into=g)', - ("at(by=lk) lands on ['g'], which the expression already carries",), + ("at(by=lk) groups by ['g'], which the expression already carries",), id='a-read-onto-a-dim-the-predicate-carries', ), pytest.param( @@ -1159,8 +1159,8 @@ class TestRulesDecidedWithoutData: 'variables.q.dims': ['g', 'h', 'z'], 'objective': {'expression': 'sum(sum(q, by=lk, over=g, into=h))'}, }, - ("sum(by=lk) lands on ['h'], which the expression already carries",), - id='landing-on-a-dim-the-operand-carries', + ("sum(by=lk) groups by ['h'], which the expression already carries",), + id='grouping-by-a-dim-the-operand-carries', ), pytest.param( {'objective': {'expression': 'sum(sum(p, by=lk, over=z, into=h))'}}, @@ -1259,8 +1259,8 @@ class TestRulesDecidedWithoutData: 'objective': {'expression': 'sum(at(r, by=rel, over=h, into=g))'}, }, ( - "at reads one value per coordinate, and 'rel' is not single-valued in ['h'] at the columns " - "the call lands on (['g'])", + "at reads one row per group, and grouping 'rel' by ['g'] leaves several rows in a group", + "its key is ['g', 'h']", ), id='at-through-a-bare-relation', ), @@ -1272,10 +1272,10 @@ class TestRulesDecidedWithoutData: }, ( "into=['z'] names ['z'], which the key of 'lz' does not hold", - "A read lands on the key it reads at, ['g']", - "Land on the key, or sum toward ['z']", + "A lookup reads one row per key, ['g']", + "Group by the key, or sum toward ['z']", ), - id='a-read-landing-on-a-value-column', + id='a-lookup-grouping-by-a-value-column', ), pytest.param( { @@ -1284,16 +1284,16 @@ class TestRulesDecidedWithoutData: 'objective': {'expression': 'sum(at(r, by=lz, over=h, into=[g, z]))'}, }, ("into=['g', 'z'] names ['z'], which the key of 'lz' does not hold",), - id='a-read-landing-on-the-key-and-a-value-column', + id='a-lookup-grouping-by-the-key-and-a-value-column', ), pytest.param( {'objective': {'expression': 'sum(sum(q, by=lk, over=h, into=g))'}}, ( - "this sum lands on the key ['g']", - 'that is a read, which is', + "the columns this sum groups by, ['g'], hold the whole key ['g']", + 'that is a join with no group-by, which is', "at(..., by=lk, over=['h'], into=['g'])", ), - id='a-sum-that-lands-on-the-key-is-a-read', + id='a-sum-grouped-by-the-whole-key-is-a-lookup', ), pytest.param( { @@ -1465,8 +1465,8 @@ class TestRulesDecidedWithoutData: ), pytest.param( {'objective': {'expression': 'sum(at(c, by=lk, over=h, into=g))'}}, - ("at(by=lk) reads through ['h'], which the expression does not carry (dims ['g'])",), - id='a-read-whose-operand-lacks-the-column-it-reads-through', + ("at(by=lk) joins on ['h'], which the expression does not carry (dims ['g'])",), + id='a-lookup-whose-operand-lacks-the-column-it-joins-on', ), pytest.param( { @@ -1515,15 +1515,15 @@ def test_a_rule_decided_without_data(self, patch, fragments): for fragment in fragments: assert fragment in message - def test_the_at_a_sum_landing_on_the_key_names_is_one_the_language_takes(self): + def test_the_at_a_sum_grouped_by_the_whole_key_names_is_one_the_language_takes(self): """A refusal that names a call is holding out a rewrite, so the rewrite has to load. - Was: the message swapped the direction's ends, answering a sum refused - for landing on the key with `at(..., over=, into=)` — which - `at` refuses in turn, for reading a column that is not single-valued - at the one the operand fixes. Both operators take `over=` as the - column the read consumes, so the rewrite is the author's own spelling - with `at` in place of `sum`. + Was: the message swapped the join's ends, answering a sum refused + for grouping by the whole key with `at(..., over=, into=)` — + which `at` refuses in turn, for grouping in a way that leaves several + rows per group. Both operators take `over=` as the columns joined on + and dropped, so the rewrite is the author's own spelling with `at` in + place of `sum`. Reading the call out of the message rather than restating it is the point: a fragment can agree with a message that names a call nothing diff --git a/tests/typesetting/golden/latex.out b/tests/typesetting/golden/latex.out index acebb2d1..abf343b5 100644 --- a/tests/typesetting/golden/latex.out +++ b/tests/typesetting/golden/latex.out @@ -101,9 +101,9 @@ \text{window} && \sum_{t' \in \mathcal{T} \,:\, 0 \le t - t' < 3} \mathit{on}_{t',g} & \le \mathit{units}_{g} && \forall\, t \in \mathcal{T},\ g \in \mathcal{G} \\ \text{history} && \sum_{t' \in \mathcal{T} \,:\, 0 \le t \ominus t' < \mathrm{min\_up}} \mathit{on}_{t',g} & \le \mathit{units}_{g} && \forall\, t \in \mathcal{T},\ g \in \mathcal{G} \\ \text{seasonal\_window} && \sum_{t' \in \mathcal{T} \,:\, 0 \le t -^{\mathrm{season\_of}(t)} t' < 3} \mathit{on}_{t',g} & \le \mathit{units}_{g} && \forall\, t \in \mathcal{T},\ g \in \mathcal{G} \\ -\text{pullback} && \mathit{spill}_{t} & \le \mathrm{zone\_cap}_{\mathrm{zone\_of}(b)} && \forall\, t \in \mathcal{T},\ b \in \mathcal{B} \\ +\text{lookup} && \mathit{spill}_{t} & \le \mathrm{zone\_cap}_{\mathrm{zone\_of}(b)} && \forall\, t \in \mathcal{T},\ b \in \mathcal{B} \\ \text{grouped\_once} && \sum_{g \in \mathcal{G} \,:\, \mathrm{gen\_bt.bus}(g) = b \wedge \mathrm{gen\_bt.technology}(g) = e} p_{t,g} & \le \mathrm{tech\_cap}_{b,e} && \forall\, t \in \mathcal{T},\ b \in \mathcal{B},\ e \in \mathcal{E} \\ -\text{pulled\_back\_once} && \mathit{units}_{g} & \le \mathrm{tech\_cap}_{\mathrm{gen\_bt.bus}(g),\mathrm{gen\_bt.technology}(g)} && \forall\, g \in \mathcal{G} \\ +\text{looked\_up\_once} && \mathit{units}_{g} & \le \mathrm{tech\_cap}_{\mathrm{gen\_bt.bus}(g),\mathrm{gen\_bt.technology}(g)} && \forall\, g \in \mathcal{G} \\ \text{within\_bus} && \mathit{units}_{g} & \le \mathit{units}_{g \boxminus_{0}^{\mathrm{gen\_bt.bus}(g)} 1} && \forall\, g \in \mathcal{G} \,:\, \mathrm{pos}_{\left( \mathrm{gen\_bt.bus}(g),\ \mathrm{gen\_bt.technology}(g) \right)}(g) = 0 \\ \text{relational} && \sum_{g \in \mathcal{G} \,:\, \left( g,\ b \right) \in \mathrm{connection}} p_{t,g} & \le \mathrm{load}_{t,b} && \forall\, t \in \mathcal{T},\ b \in \mathcal{B} \\ \text{connected} && p_{t,g} & \le \mathrm{load}_{t,b} && \forall\, t \in \mathcal{T},\ g \in \mathcal{G},\ b \in \mathcal{B} \,:\, \left( g,\ b \right) \in \mathrm{connection} \\ @@ -111,7 +111,7 @@ \text{zonal} && \sum_{g \in \mathcal{G} \,:\, \mathrm{gen\_zone}(g,\ t) = z} p_{t,g} & \le \mathrm{zone\_cap}_{z} && \forall\, t \in \mathcal{T},\ z \in \mathcal{Z} \\ \text{zonal\_history} && \sum_{t \in \mathcal{T} \,:\, \mathrm{gen\_zone}(g,\ t) = z} p_{t,g} & \le \mathrm{zone\_cap}_{z} && \forall\, g \in \mathcal{G},\ z \in \mathcal{Z} \\ \text{zonal\_membership} && \sum_{g \in \mathcal{G} \,:\, \mathrm{gen\_zone}(g,\ t) \text{ is defined}} \mathit{units}_{g} & \le \mathrm{budget} && \forall\, t \in \mathcal{T} \\ -\text{zonal\_pullback} && p_{t,g} & \le \mathit{spill}_{t} \cdot \mathrm{zone\_cap}_{\mathrm{gen\_zone}(g,\ t)} && \forall\, t \in \mathcal{T},\ g \in \mathcal{G} \,:\, \mathrm{gen\_zone}(g,\ t) = \text{'}\mathrm{north}\text{'} \wedge \mathrm{pos}_{\mathrm{gen\_zone}(g,\ t)}(g) = 0 \\ +\text{zonal\_lookup} && p_{t,g} & \le \mathit{spill}_{t} \cdot \mathrm{zone\_cap}_{\mathrm{gen\_zone}(g,\ t)} && \forall\, t \in \mathcal{T},\ g \in \mathcal{G} \,:\, \mathrm{gen\_zone}(g,\ t) = \text{'}\mathrm{north}\text{'} \wedge \mathrm{pos}_{\mathrm{gen\_zone}(g,\ t)}(g) = 0 \\ \text{arithmetic} && \sum_{g \in \mathcal{G}} \left( \frac{p_{t,g}}{2} - \mathrm{cost}_{g} + 10^{-5} \cdot p_{t,g} + 2.5 \times 10^{-7} \cdot \mathrm{cost}_{g} + 0.5 \cdot p_{t,g} \right) & \ge -\left( \sum_{g \in \mathcal{G}} p_{t,g} \right) \cdot \left( -3 \right) && \forall\, t \in \mathcal{T} \\ \text{total} && \sum_{t \in \mathcal{T},\ g \in \mathcal{G}} p_{t,g} & \le \mathrm{budget} \\ \text{scalar} && \mathit{units}_{g} & \le \mathrm{budget} && \forall\, g \in \mathcal{G} \,:\, \mathrm{cost}_{g} \text{ is defined} \\ diff --git a/tests/typesetting/golden/markdown.out b/tests/typesetting/golden/markdown.out index 672578c3..7bb2241f 100644 --- a/tests/typesetting/golden/markdown.out +++ b/tests/typesetting/golden/markdown.out @@ -176,7 +176,7 @@ p_{t,g} \le p_{t \boxminus_{0}^{\mathrm{season\_of}(t)} 1,g} \qquad \forall\, t \sum_{t' \in \mathcal{T} \,:\, 0 \le t -^{\mathrm{season\_of}(t)} t' < 3} \mathit{on}_{t',g} \le \mathit{units}_{g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} ``` -**`pullback`** +**`lookup`** ```math \mathit{spill}_{t} \le \mathrm{zone\_cap}_{\mathrm{zone\_of}(b)} \qquad \forall\, t \in \mathcal{T},\ b \in \mathcal{B} @@ -188,7 +188,7 @@ p_{t,g} \le p_{t \boxminus_{0}^{\mathrm{season\_of}(t)} 1,g} \qquad \forall\, t \sum_{g \in \mathcal{G} \,:\, \mathrm{gen\_bt.bus}(g) = b \wedge \mathrm{gen\_bt.technology}(g) = e} p_{t,g} \le \mathrm{tech\_cap}_{b,e} \qquad \forall\, t \in \mathcal{T},\ b \in \mathcal{B},\ e \in \mathcal{E} ``` -**`pulled_back_once`** +**`looked_up_once`** ```math \mathit{units}_{g} \le \mathrm{tech\_cap}_{\mathrm{gen\_bt.bus}(g),\mathrm{gen\_bt.technology}(g)} \qquad \forall\, g \in \mathcal{G} @@ -236,7 +236,7 @@ p_{t,g} \le \mathrm{load}_{t,b} \qquad \forall\, t \in \mathcal{T},\ g \in \math \sum_{g \in \mathcal{G} \,:\, \mathrm{gen\_zone}(g,\ t) \text{ is defined}} \mathit{units}_{g} \le \mathrm{budget} \qquad \forall\, t \in \mathcal{T} ``` -**`zonal_pullback`** +**`zonal_lookup`** ```math p_{t,g} \le \mathit{spill}_{t} \cdot \mathrm{zone\_cap}_{\mathrm{gen\_zone}(g,\ t)} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} \,:\, \mathrm{gen\_zone}(g,\ t) = \text{'}\mathrm{north}\text{'} \wedge \mathrm{pos}_{\mathrm{gen\_zone}(g,\ t)}(g) = 0 diff --git a/tests/typesetting/golden/model.yaml b/tests/typesetting/golden/model.yaml index 9f406920..e68e2108 100644 --- a/tests/typesetting/golden/model.yaml +++ b/tests/typesetting/golden/model.yaml @@ -26,7 +26,7 @@ relations: zone_of: { key: bus, values: zone } area_of: { key: bus, values: zone } # a second map into the same set, to compare against season_of: { key: snapshot, values: season } - gen_zone: { key: [generator, snapshot], values: zone } # a map keyed by two dimensions: a call consumes one and joins on the other + gen_zone: { key: [generator, snapshot], values: zone } # a map keyed by two dimensions: a call sums one away and joins on the other rep_of: { key: snapshot, values: { rep: snapshot } } # a map into its own dimension: the representative snapshot connection: { key: [generator, bus] } # a bare relation, with no value columns: many-to-many, read only by sum with both ends named gen_bt: { key: generator, values: [bus, technology] } # one table with two value columns, read to both at once @@ -201,13 +201,13 @@ constraints: seasonal_window: # a window partitioned by a relation: the group rides on the operator dims: [snapshot, generator] expression: sum_back(on, along=snapshot, window=3, by=season_of, within=season) <= units - pullback: # at(), which re-indexes through a relation instead of an offset + lookup: # at(), which re-indexes through a relation instead of an offset dims: [snapshot, bus] expression: spill <= at(zone_cap, by=zone_of, over=zone, into=bus) grouped_once: # one table read to two value columns: the domain carries a condition per column dims: [snapshot, bus, technology] expression: sum(p, by=gen_bt, into=[bus, technology], over=generator) <= tech_cap - pulled_back_once: # its adjoint, reading one slot through two columns of one table + looked_up_once: # the same table joined the other way, reading one slot through two columns dims: [generator] expression: units <= at(tech_cap, by=gen_bt, over=[bus, technology], into=generator) within_bus: # a partition grouped by one named value column of a two-value table, and a position within both @@ -224,16 +224,16 @@ constraints: representative: # a map into its own dimension, read both ways: the frame is unchanged and the index is primed dims: [snapshot] expression: sum(spill, by=rep_of, over=snapshot, into=rep) <= at(spill, by=rep_of, over=rep, into=snapshot) - zonal: # a grouping through a two-key map, consuming one key: the condition reads the other, and the row keeps it + zonal: # a grouping through a two-key map, summing one key away: the condition reads the other, and the row keeps it dims: [snapshot, zone] expression: sum(p, by=gen_zone, over=generator, into=zone) <= zone_cap - zonal_history: # the same table consuming its other key + zonal_history: # the same table summing its other key away dims: [generator, zone] expression: sum(p, by=gen_zone, over=snapshot, into=zone) <= zone_cap zonal_membership: # the same table read between its two key columns: no value column is read, so the domain asks only that the row is there dims: [snapshot] expression: sum(units, by=gen_zone, over=generator, into=snapshot) <= budget - zonal_pullback: # its adjoint, reading the slot the row's own snapshot puts the generator in + zonal_lookup: # the same table joined the other way, reading the slot the row's own snapshot puts the generator in dims: [snapshot, generator] where: "gen_zone == 'north' AND position(generator, by=gen_zone, within=zone) == 0" expression: p <= at(spill * zone_cap, by=gen_zone, into=generator, over=zone) diff --git a/tests/typesetting/golden/typst.out b/tests/typesetting/golden/typst.out index 68bda848..1ef2267d 100644 --- a/tests/typesetting/golden/typst.out +++ b/tests/typesetting/golden/typst.out @@ -88,9 +88,9 @@ $ upright("budgeted") & italic("spend")_(t) & <= upright("budget") & forall t in upright("window") & sum_(t' in cal(T) colon 0 <= t - t' < 3) italic("on")_(t',g) & <= italic("units")_(g) & forall t in cal(T), g in cal(G) \ upright("history") & sum_(t' in cal(T) colon 0 <= t minus.o t' < upright("min_up")) italic("on")_(t',g) & <= italic("units")_(g) & forall t in cal(T), g in cal(G) \ upright("seasonal_window") & sum_(t' in cal(T) colon 0 <= t -^(upright("season_of")(t)) t' < 3) italic("on")_(t',g) & <= italic("units")_(g) & forall t in cal(T), g in cal(G) \ - upright("pullback") & italic("spill")_(t) & <= upright("zone_cap")_(upright("zone_of")(b)) & forall t in cal(T), b in cal(B) \ + upright("lookup") & italic("spill")_(t) & <= upright("zone_cap")_(upright("zone_of")(b)) & forall t in cal(T), b in cal(B) \ upright("grouped_once") & sum_(g in cal(G) colon upright("gen_bt.bus")(g) = b and upright("gen_bt.technology")(g) = e) p_(t,g) & <= upright("tech_cap")_(b,e) & forall t in cal(T), b in cal(B), e in cal(E) \ - upright("pulled_back_once") & italic("units")_(g) & <= upright("tech_cap")_(upright("gen_bt.bus")(g),upright("gen_bt.technology")(g)) & forall g in cal(G) \ + upright("looked_up_once") & italic("units")_(g) & <= upright("tech_cap")_(upright("gen_bt.bus")(g),upright("gen_bt.technology")(g)) & forall g in cal(G) \ upright("within_bus") & italic("units")_(g) & <= italic("units")_(g minus.square_(0)^(upright("gen_bt.bus")(g)) 1) & forall g in cal(G) colon upright("pos")_((upright("gen_bt.bus")(g), upright("gen_bt.technology")(g)))(g) = 0 \ upright("relational") & sum_(g in cal(G) colon (g, b) in upright("connection")) p_(t,g) & <= upright("load")_(t,b) & forall t in cal(T), b in cal(B) \ upright("connected") & p_(t,g) & <= upright("load")_(t,b) & forall t in cal(T), g in cal(G), b in cal(B) colon (g, b) in upright("connection") \ @@ -98,7 +98,7 @@ $ upright("budgeted") & italic("spend")_(t) & <= upright("budget") & forall t in upright("zonal") & sum_(g in cal(G) colon upright("gen_zone")(g, t) = z) p_(t,g) & <= upright("zone_cap")_(z) & forall t in cal(T), z in cal(Z) \ upright("zonal_history") & sum_(t in cal(T) colon upright("gen_zone")(g, t) = z) p_(t,g) & <= upright("zone_cap")_(z) & forall g in cal(G), z in cal(Z) \ upright("zonal_membership") & sum_(g in cal(G) colon upright("gen_zone")(g, t) upright(" is defined")) italic("units")_(g) & <= upright("budget") & forall t in cal(T) \ - upright("zonal_pullback") & p_(t,g) & <= italic("spill")_(t) dot upright("zone_cap")_(upright("gen_zone")(g, t)) & forall t in cal(T), g in cal(G) colon upright("gen_zone")(g, t) = upright("'north'") and upright("pos")_(upright("gen_zone")(g, t))(g) = 0 \ + upright("zonal_lookup") & p_(t,g) & <= italic("spill")_(t) dot upright("zone_cap")_(upright("gen_zone")(g, t)) & forall t in cal(T), g in cal(G) colon upright("gen_zone")(g, t) = upright("'north'") and upright("pos")_(upright("gen_zone")(g, t))(g) = 0 \ upright("arithmetic") & sum_(g in cal(G)) (frac(p_(t,g), 2) - upright("cost")_(g) + 10^(-5) dot p_(t,g) + 2.5 times 10^(-7) dot upright("cost")_(g) + 0.5 dot p_(t,g)) & >= -(sum_(g in cal(G)) p_(t,g)) dot (-3) & forall t in cal(T) \ upright("total") & sum_(t in cal(T), g in cal(G)) p_(t,g) & <= upright("budget") \ upright("scalar") & italic("units")_(g) & <= upright("budget") & forall g in cal(G) colon upright("cost")_(g) upright(" is defined") \ diff --git a/tests/typesetting/test_golden.py b/tests/typesetting/test_golden.py index dc75b4d7..07870d88 100644 --- a/tests/typesetting/test_golden.py +++ b/tests/typesetting/test_golden.py @@ -16,7 +16,7 @@ import pytest from math_spec.operators import BUILTIN_NAMES -from math_spec.program import Dual, Expression, GroupSum, Named, Predicate, Pullback, Sum, Translate, WindowSum +from math_spec.program import Dual, Expression, Join, Named, Predicate, Sum, Translate, WindowSum from math_spec.typesetting import FORMATS, to_latex, typeset, walk from math_spec.typesetting.format import OPERATOR_NAMES from math_spec.validation import to_spec @@ -144,12 +144,12 @@ def _rendered_trees() -> Iterator[object]: #: A dataclass the walk steps *through* rather than renders: a region has no -#: branch of its own — its ``when`` and ``value`` do — a direction and the -#: relation it reads are the facts a node carries rather than nodes, and a -#: ``Mask`` is the wrapper a leaf carries a predicate in. None is a member of -#: any node union, so they are subtracted from what the tree walk finds rather -#: than added to what the vocabulary declares. -CARRIERS = {'Region', 'Direction', 'Mask', 'Partition', 'RelationDeclaration'} +#: branch of its own — its ``when`` and ``value`` do — the columns a join +#: names and the relation it reads are the facts a node carries rather than +#: nodes, and a ``Mask`` is the wrapper a leaf carries a predicate in. None is a +#: member of any node union, so they are subtracted from what the tree walk +#: finds rather than added to what the vocabulary declares. +CARRIERS = {'Region', 'JoinColumns', 'Mask', 'Partition', 'RelationDeclaration'} def test_the_golden_model_carries_every_node_kind_the_walk_renders(): @@ -175,9 +175,10 @@ def test_the_golden_model_calls_every_operator_in_the_language(): Each operator resolves to the node it is, so the census counts the nodes by the verb the file writes them with. """ - verbs = {Sum: 'sum', GroupSum: 'sum', Pullback: 'at', Translate: 'shift', WindowSum: 'sum_back', Dual: 'dual'} + verbs = {Sum: 'sum', Translate: 'shift', WindowSum: 'sum_back', Dual: 'dual'} nodes = [node for tree in _rendered_trees() for node in _nodes(tree)] calls = {verb for node in nodes for kind, verb in verbs.items() if isinstance(node, kind)} + calls |= {'at' for node in nodes if isinstance(node, Join) and not node.columns.axes} assert calls == BUILTIN_NAMES, ( f'tests/typesetting/golden/model.yaml never calls {sorted(BUILTIN_NAMES - calls)}. ' f'An operator with no case here renders untested.' diff --git a/tests/typesetting/test_walk.py b/tests/typesetting/test_walk.py index 1b4a78e8..072d810e 100644 --- a/tests/typesetting/test_walk.py +++ b/tests/typesetting/test_walk.py @@ -175,7 +175,7 @@ def test_a_fill_and_a_group_take_the_operators_two_slots(name: FormatName, fmt: @EVERY_FORMAT -def test_a_translation_under_a_pullback_survives_it(name: FormatName, fmt: Format): +def test_a_translation_under_a_lookup_survives_it(name: FormatName, fmt: Format): """``at`` and ``shift`` both re-index at the leaf, and the leaf has one subscript. Whoever wrote it last used to win: ``at(shift(cap, along=period, offset=1, @@ -200,7 +200,7 @@ def test_a_translation_under_a_pullback_survives_it(name: FormatName, fmt: Forma } text = typeset(model, name, legend=False) assert fmt.operators['edge_minus'] in text, 'the shift under the at was dropped from the subscript' - assert fmt.apply(fmt.upright('period_of'), 't') in text, 'the pullback itself was dropped' + assert fmt.apply(fmt.upright('period_of'), 't') in text, 'the lookup itself was dropped' @EVERY_FORMAT