feat(language): a where may compare arithmetic over parameters

docs/reference/language/expressions.md

@@ -157,10 +157,11 @@ A `where:` is a boolean mask, and true means "this coordinate exists".
 ```text
 where_expr ::= atom | "NOT" where_expr | where_expr ("AND"|"OR") where_expr
             |  "(" where_expr ")"
-atom       ::= NAME | NAME COMPARATOR value | POSITION COMPARATOR INTEGER
-            |  "True" | "False"
+atom       ::= NAME | NAME COMPARATOR value | expression COMPARATOR expression
+            |  POSITION COMPARATOR INTEGER | "True" | "False"
 COMPARATOR ::= "<=" | ">=" | "==" | "!=" | "<" | ">"
 value      ::= NUMBER | QUOTED | NAME_OR_STRING
+expression ::= the arithmetic grammar above, with no variable and no dual in it
 POSITION   ::= "position" "(" NAME [ "," "by" "=" NAME ] ")"
 QUOTED     ::= "'" chars "'" | '"' chars '"'
 ```
@@ -175,6 +176,7 @@ QUOTED     ::= "'" chars "'" | '"' chars '"'
 | `name OP value`                           | dimension                                | A filter on the frame's own coordinate column                                                                                                                                                                                                                                                       |
 | `name OP value`, `name.col OP value`      | relation                                 | A filter on a value column of a keyed relation, read at its key, so the key's dimensions have to be in the frame. Name the column where the key determines several. A null compares false                                                                                                           |
 | `name OP name`, `name.a OP name.b`        | two relation columns                     | Legal only where both relations are keyed over the same dimensions and both columns are over one dimension. `ends.bus0 != ends.bus1` excludes a self-loop                                                                                                                                           |
+| `expression OP expression`                | arithmetic over parameters               | Coordinate by coordinate, over every dimension either side carries. A macro and a named expression expand as in an expression, and every operator keeps its rule, so a `shift` names its `edge=`. A side with no value at a coordinate compares false                                               |
 | `position(name) OP i`                     | dimension                                | Where the row sits along the dimension's own order. `0` is first, and a negative number counts from the end                                                                                                                                                                                         |
 | `position(name, by=relation[, within=c])` | a dimension and a relation keyed over it | The same, counted within each group the relation's value columns make                                                                                                                                                                                                                               |
 | `AND` `OR` `NOT`                          | —                                        | Case-insensitive. `NOT` binds tighter than `AND`, and `AND` tighter than `OR`                                                                                                                                                                                                                       |
@@ -218,6 +220,53 @@ dimension. Keyed alike, they are two columns of one key table, so the comparison
 filters that table rather than joining two. Over one dimension they draw from one
 label set, so a match is possible at all.
 
+### Arithmetic in a comparison
+
+Either side of a comparison may be an expression: `p_min <= 0.5 * p_max`,
+`sum(p_max, over=generator) >= peak`, `p_max <= at(bus_cap, by=bus_of)`. The
+side is read exactly as an [expression](#expressions) is, so a macro and a
+named expression expand into it and every operator keeps its own rule. Two
+things an expression may carry are refused here, because a mask is built before
+either exists: a variable, and a `dual()`.
+
+A comparison of expressions is checked over every dimension either side
+carries. A side whose value is absent at a coordinate compares false there, as
+a null does in every other comparison; under a summing operator the absent
+term is one fewer. A `shift` says what its vacated positions hold, as it does
+everywhere, so a comparison against the previous row names an `edge=` and a
+`position()` term keeps the first row out:
+
+```yaml
+dimensions:
+  snapshot: { dtype: int }
+parameters:
+  load: { dims: [snapshot] }
+  ramp: { dims: [] }
+variables:
+  shed: { dims: [snapshot], bounds: { lower: 0 } }
+constraints:
+  shed_when_load_jumps:
+    dims: [snapshot]
+    where: "load - shift(load, along=snapshot, offset=1, edge=0) > ramp AND position(snapshot) > 0"
+    expression: shed >= load - ramp
+```
+
+A case `when:` may not compare expressions. The loader proves the cases of a
+[`cases:` block](#the-rules-that-keep-the-cases-apart) apart at load: no two of
+them may claim one coordinate. It proves that by trying every value the masks
+name. A comparison of expressions names no value, because only the data decides
+whether `c > 2 * k` holds. There is nothing to try, so the loader refuses the
+block:
+
+> `Named expression 'e'`: cases `wide` and `narrow` cannot be told apart before
+> the data arrives: it compares expressions, whose values only the data decides
+> — compare one parameter against a literal, or precompute the test as a boolean
+> parameter and test that. Two cases claiming one coordinate would give it two
+> values, so this is refused the way a proven overlap is.
+
+A variable's `where` and a constraint's `where` are not held to this, because
+neither is proved apart from anything.
+
 ### `position()`
 
 `position(dim)` is where the row sits along the dimension's own order, which is

docs/reference/language/reading.md

@@ -109,6 +109,11 @@ questions that every engine would otherwise work out for itself:
 - `.atoms` gives its leaves, with the connectives removed.
 - `.dims` gives the dimensions the mask is read at.
 
+A comparison of expressions arrives as an `ExpressionComparisonNode`, whose two
+sides are program expressions like a constraint's, and whose `dims` are every
+dimension either side carries. Its `names_read` are every parameter and lookup
+the sides read, the lookup a grouping joins through included.
+
 A predicate you build yourself answers the same four questions: wrap it in `Mask`,
 or build it there with `~`, `&` and `|`. A mask folds as it is built: a double
 negation cancels, and a `True` or `False` is absorbed rather than buried in the

docs/reference/notation.md

@@ -119,6 +119,7 @@ parameters:
 
 | Symbol | Meaning |
 |---|---|
+| $`\mathrm{spend}^{\mathrm{cap}}`$ | `spend_cap` over $`\mathcal{G}`$ |
 | $`\mathit{spend}`$ | `spend` over $`\mathcal{T}`$ — what a snapshot's dispatch costs |
 | $`\mathit{lcoe}`$ | `lcoe` (scalar) |
 | $`\mathit{marginal\_price}`$ | `marginal_price` over $`\mathcal{T} \times \mathcal{B}`$ |
@@ -716,8 +717,80 @@ never:
 \mathit{slack}_{t} \ge 0 \qquad \forall\, t \in \mathcal{T} \,:\, \bot
 ```
 
+#### `margin`
+
+a mask comparing two expressions, which prints as the arithmetic it is
+
+```yaml
+margin:
+  dims: [snapshot, generator]
+  where: "p_max - p_min > cost / 2"
+  expression: p <= p_max
+```
+
+```math
+p_{t,g} \le \mathrm{p}^{\mathrm{max}}_{g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} \,:\, \mathrm{p}^{\mathrm{max}}_{g} - \mathrm{p}^{\mathrm{min}}_{g} > \frac{\mathrm{cost}_{g}}{2}
+```
+
+#### `ramped`
+
+a translation under a comparison names its edge, a pullback reads through a lookup, and the position keeps the vacated row out
+
+```yaml
+ramped:
+  dims: [snapshot, bus]
+  where: "load - shift(load, along=snapshot, offset=1, edge=0) <= at(zone_cap, by=zone_of) AND position(snapshot) > 0"
+  expression: slack <= load
+```
+
+```math
+\mathit{slack}_{t} \le \mathrm{load}_{t,b} \qquad \forall\, t \in \mathcal{T},\ b \in \mathcal{B} \,:\, \mathrm{load}_{t,b} - \mathrm{load}_{t \boxminus_{0} 1,b} \le \mathrm{zone\_cap}_{\mathrm{zone\_of}(b)} \wedge \mathrm{pos}(t) > 0
+```
+
+#### `covered`
+
+a reduction on a side of a scalar mask, so nothing is left to quantify
+
+```yaml
+covered:
+  dims: []
+  where: "sum(p_max, over=generator) >= budget"
+  expression: sum(p) <= budget
+```
+
+```math
+\sum_{t \in \mathcal{T},\ g \in \mathcal{G}} p_{t,g} \le \mathrm{budget} \qquad \text{where } \sum_{g \in \mathcal{G}} \mathrm{p}^{\mathrm{max}}_{g} \ge \mathrm{budget}
+```
+
+#### `capped`
+
+an expressions: entry on a side, read by the name the file gave it
+
+```yaml
+capped:
+  dims: [snapshot, generator]
+  where: "spend_cap > 0 OR NOT is_flexible"
+  expression: p <= p_max
+```
+
+```math
+p_{t,g} \le \mathrm{p}^{\mathrm{max}}_{g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} \,:\, \mathrm{spend}^{\mathrm{cap}}_{g} > 0 \vee \neg \mathrm{is\_flexible}_{g}
+```
+
 ### Definitions
 
+#### `spend_cap`
+
+a data-only entry, so a where may compare it
+
+```yaml
+spend_cap: cost * 2
+```
+
+```math
+\mathrm{spend}^{\mathrm{cap}}_{g} = \mathrm{cost}_{g} \cdot 2 \qquad \forall\, g \in \mathcal{G}
+```
+
 #### `spend`
 
 a plain named expression: its symbol prints where it is used, its body once as a definition

src/math_spec/_expression_parser.py

@@ -353,8 +353,12 @@ def with_children(node: ArithmeticNode, recurse: Callable[[ArithmeticNode], Arit
 # ---------------------------------------------------------------------------
 
 
-def _build_grammar() -> pp.ParserElement:
-    """``inf`` is a ``pp.Keyword`` rather than a ``pp.Literal``, which would match the prefix of ``inflow``."""
+def _build_grammar() -> tuple[pp.ParserElement, pp.ParserElement]:
+    """The arithmetic grammar, and the expression grammar that puts one comparison over it.
+
+    ``inf`` is a ``pp.Keyword`` rather than a ``pp.Literal``, which would
+    match the prefix of ``inflow``.
+    """
     arith = pp.Forward()
 
     inf_literal = (pp.Keyword('.inf') | pp.Keyword('inf')).set_parse_action(lambda: NumberNode(float('inf')))
@@ -390,9 +394,10 @@ def _build_grammar() -> pp.ParserElement:
     arith <<= add_sub
 
     comparator = pp.one_of(list(get_args(ComparisonOperator)))
-    return (arith + pp.Optional(comparator + arith)).set_parse_action(
+    expression = (arith + pp.Optional(comparator + arith)).set_parse_action(
         lambda t: ComparisonNode(t[1], t[0], t[2]) if len(t) == 3 else t[0]
     )
+    return arith, expression
 
 
 def _make_func_call(tokens: pp.ParseResults) -> FunctionCallNode:
@@ -425,7 +430,9 @@ def _make_power(tokens: pp.ParseResults) -> Any:
     return items[0] if len(items) == 1 else BinaryOperatorNode('**', items[0], items[2])
 
 
-_GRAMMAR = _build_grammar()
+#: The arithmetic half on its own, for the where grammar to put a predicate's
+#: comparator over — one grammar for what a side may say, wherever it stands.
+ARITHMETIC, _GRAMMAR = _build_grammar()
 
 
 #: How deep a tree the language admits. Every pass over an expression recurses,

src/math_spec/_where_parser.py

@@ -15,12 +15,23 @@
 
 import pyparsing as pp
 
-from math_spec._expression_parser import NAME, REAL, parse_text
+from math_spec._expression_parser import (
+    ARITHMETIC,
+    NAME,
+    REAL,
+    FunctionCallNode,
+    NameNode,
+    NumberNode,
+    UnaryOperatorNode,
+    children,
+    parse_text,
+)
 from math_spec.program import AndNode, BooleanLiteralNode, NotNode, OrNode, PredicateOperator, where_children
 
 if TYPE_CHECKING:
     from collections.abc import Callable
 
+    from math_spec._expression_parser import ArithmeticNode
     from math_spec.program import WhereNode
 
 # ---------------------------------------------------------------------------
@@ -49,6 +60,19 @@ class UnresolvedComparisonNode:
     quoted: bool = False
 
 
+@dataclass(frozen=True)
+class UnresolvedExpressionComparisonNode:
+    """``expression <op> expression``, both sides still the bare parse — ``resolution.py`` types and judges them.
+
+    The grammar reaches for this only where a side is more than one name or
+    literal, so the simpler forms keep their own nodes and their own rules.
+    """
+
+    left: ArithmeticNode
+    op: PredicateOperator
+    right: ArithmeticNode
+
+
 @dataclass(frozen=True)
 class UnresolvedPositionNode:
     """``position(dim[, by=relation[, within=columns]]) <op> i`` before the names are checked; ``resolution.py`` types it."""
@@ -62,7 +86,9 @@ class UnresolvedPositionNode:
 
 #: What resolution rewrites away on the where side — the three nodes whose
 #: left-hand side is still a name the schema has not been asked about.
-UnresolvedWhereNode = UnresolvedNameNode | UnresolvedComparisonNode | UnresolvedPositionNode
+UnresolvedWhereNode = (
+    UnresolvedNameNode | UnresolvedComparisonNode | UnresolvedExpressionComparisonNode | UnresolvedPositionNode
+)
 
 
 # ---------------------------------------------------------------------------
@@ -84,6 +110,31 @@ def _position_comparison(tokens: pp.ParseResults) -> UnresolvedPositionNode:
     return UnresolvedPositionNode(str(dimension), op, at, by, into)
 
 
+def _is_plain(node: ArithmeticNode) -> bool:
+    """Whether *node* is one name or one signed number — a side the simpler comparison forms own."""
+    if isinstance(node, UnaryOperatorNode):
+        return isinstance(node.operand, NumberNode)
+    return isinstance(node, NameNode | NumberNode)
+
+
+def _reads_arithmetic(tokens: pp.ParseResults) -> bool:
+    """Whether a comparison needs the expression form at all.
+
+    Two plain sides are ``name <op> literal`` or ``name <op> name``, and a
+    ``position(...)`` call against a plain side is the position form; each of
+    those has a node of its own, so this form stands aside for them.
+    """
+    left, _, right = tokens
+    if _is_plain(left) and _is_plain(right):
+        return False
+    return not (isinstance(left, FunctionCallNode) and left.name == 'position' and _is_plain(right))
+
+
+def _expression_comparison(tokens: pp.ParseResults) -> UnresolvedExpressionComparisonNode:
+    left, op, right = tokens
+    return UnresolvedExpressionComparisonNode(left, op, right)
+
+
 def _comparison(tokens: pp.ParseResults) -> UnresolvedComparisonNode:
     """``name <op> literal`` off the tokens the grammar captured, the quoted marker turned into a flag."""
     name, op, value = tokens
@@ -95,8 +146,11 @@ def _build_where_grammar() -> pp.ParserElement:
     """Build the pyparsing grammar for where strings.
 
     Both quote characters are accepted because YAML already owns one of them.
-    ``NOT`` binds tightest, then ``AND``, then ``OR``. ``position(...)`` leads
-    the alternation, since ``position`` would otherwise be read as a bare name.
+    ``NOT`` binds tightest, then ``AND``, then ``OR``. The three comparison
+    forms are matched longest-first, so ``p > 2 * q`` is not cut short at
+    ``p > 2``; the expression form stands aside for the two plain shapes
+    (:func:`_reads_arithmetic`), so ``p > 0`` keeps the node its dtype rule
+    is written for.
     """
     where_expr = pp.Forward()
 
@@ -128,14 +182,16 @@ def _build_where_grammar() -> pp.ParserElement:
     position_comparison = (position_call + comparator + position).set_parse_action(_position_comparison)
 
     comparison = (column + comparator + (number | quoted | column)).set_parse_action(_comparison)
+    expression_comparison = (
+        (ARITHMETIC + comparator + ARITHMETIC).add_condition(_reads_arithmetic).add_parse_action(_expression_comparison)
+    )
     # pyrefly: ignore[implicit-any-lambda]
     existence = name.copy().set_parse_action(lambda t: UnresolvedNameNode(t[0]))
 
     atom = (
         true_lit
         | false_lit
-        | position_comparison
-        | comparison
+        | (position_comparison ^ comparison ^ expression_comparison)
         | existence
         | (pp.Suppress('(') + where_expr + pp.Suppress(')'))
     )
@@ -199,6 +255,17 @@ def _named_rewrite(text: str, loc: int) -> str | None:
 )
 
 
+def _nested(node: Any) -> tuple[Any, ...]:
+    """What a where string nests through: a connective's operands, and the arithmetic under a comparison of expressions."""
+    if isinstance(node, UnresolvedExpressionComparisonNode):
+        return (node.left, node.right)
+    if isinstance(node, UnresolvedWhereNode):
+        return ()
+    if isinstance(node, AndNode | OrNode | NotNode | BooleanLiteralNode):
+        return where_children(node)
+    return children(node)
+
+
 @lru_cache(maxsize=4096)
 def parse_where(text: str) -> WhereNode | UnresolvedWhereNode:
     """Parse a where string into an AST, its leaves still unresolved.
@@ -215,5 +282,5 @@ def parse_where(text: str) -> WhereNode | UnresolvedWhereNode:
     """
     return cast(
         'WhereNode | UnresolvedWhereNode',
-        parse_text(_WHERE_GRAMMAR, text, 'where string', _named_rewrite, where_children, _DEEP_REWRITE),
+        parse_text(_WHERE_GRAMMAR, text, 'where string', _named_rewrite, _nested, _DEEP_REWRITE),
     )

src/math_spec/dimensions.py

@@ -40,8 +40,10 @@
 from math_spec.errors import DimensionError
 from math_spec.operators import BUILTINS
 from math_spec.program import (
+    ArithmeticComparisonNode,
     DimensionComparisonNode,
     DimensionPositionNode,
+    ExpressionComparisonNode,
     Mask,
     ParameterComparisonNode,
     ParameterDefinedNode,
@@ -521,17 +523,19 @@ def _check_where_dims(
             continue
         match atom:
             case ParameterDefinedNode() | ParameterComparisonNode():
-                noun = 'parameter'
+                leaf = f"where-parameter '{atom.name}'"
             case VariableDefinedNode():
-                noun = 'variable'
+                leaf = f"where-variable '{atom.name}'"
             case DimensionComparisonNode() | DimensionPositionNode():
-                noun = 'dimension'
+                leaf = f"where-dimension '{atom.name}'"
             case RelationComparisonNode() | RelationPairComparisonNode() | RelationDefinedNode():
-                noun = 'relation'
+                leaf = f"where-relation '{atom.name}'"
+            case ArithmeticComparisonNode() | ExpressionComparisonNode():
+                leaf = 'a where-comparison of expressions'
             case _:
                 assert_never(atom)
         raise DimensionError(
-            f"{context}: where-{noun} '{atom.name}' reads dims {outside} outside the frame {sorted(frame)}. "
+            f'{context}: {leaf} reads dims {outside} outside the frame {sorted(frame)}. '
             f'Reducing a mask over an unlisted dim would silently widen it — add the dim to dims:, '
             f'or test a name the frame carries.'
         )