feat(language): a decided variable becomes a supplied number, so a subproblem is a call rather than a second file

docs/reference/language/declarations.md

@@ -20,7 +20,10 @@ characters rather than as a symbol. Write the thing rather than its symbol —
 ## `parameters`
 
 Declared shape only; the numbers bind by name at run time, in whatever
-consumes the AST.
+consumes the AST. What that binding may not decide for itself — where a
+dimension's members come from, the order they stand in, and that a table
+carries each coordinate at most once — is in
+[dimensions](dimensions.md).
 
 ```yaml
 dimensions:
@@ -36,8 +39,30 @@ parameters:
 | ------------- | ------------------------------------------------------------ | --------------- |
 | `dims`        | required — the dimensions it is indexed by; `[]` is a scalar |                 |
 | `dtype`       | `float`, `int`, `bool`, `str`                                | default `float` |
+| `coverage`    | `total`, `masked`                                            | default `total` |
 | `description` | free text                                                    | default `null`  |
 
+**`coverage` says whether a missing row was meant.** A table short of a
+coordinate and a table that never had one look identical in the data, and they
+mean opposite things: `total` is the claim that every coordinate the `dims`
+reach has a value, so a row that went missing in preparation is an error rather
+than a mask; `masked` is the file saying the gap is the point — the parameter
+_is_ a mask, and a coordinate it leaves out is [absence](absence.md).
+
+```yaml
+parameters:
+  cost: { dims: [generator] } # total: every generator has one
+  ramp_limit: { dims: [generator], coverage: masked } # no row means no limit
+```
+
+Without it the reading is a consumer's to pick, and two consumers picking
+differently would build different models from one file and one table — so the
+declaration says it and no consumer guesses. **The default is `total`** because
+that is what the other rules already assume: a bound and a divisor
+[refuse absence outright](absence.md), so a parameter reaching either must
+cover its rows, and a file declaring `masked` in those positions is a load
+error naming the rewrite.
+
 **`dtype` is a claim about the values, and the column has to be it.** It
 decides four things — whether the name is a value in an
 [expression](expressions.md) at all, what a `where` comparison is checked

docs/reference/language/dimensions.md

@@ -32,8 +32,43 @@ in step by hand.
 **One master coordinate set per dimension, resolved before any data binds.**
 Every parameter is reindexed onto it, so two tables that disagree about which
 snapshots exist is an error at load time rather than a silently truncated
-model. Where the coordinates come from, and in what order, is settled when
-data is bound — which this package declares the shape of and does not do.
+model. Which coordinates those are, and in what order they stand, is data's to
+say — and the three rules below say how it says it.
+
+### Binding is the language's, even though the data is not
+
+The file declares an axis; the data supplies its members. Between those two
+sentences sit three facts that decide **which model a file and a table make
+together** — and a consumer answering any of them differently would build a
+different model from the same two inputs. So they are the language's, and a
+consumer implements them rather than choosing them.
+
+**The dimension's own source supplies its members.** They are read from the key
+named after the dimension, and from nothing else: a parameter's table is read
+for values, never for labels, and a lookup's map is not a claim about which
+members exist. A dimension nothing supplies is an error naming it, not an empty
+axis — an axis with no members would delete every row indexed by it, silently.
+
+**Their order is the order that source gives them**, first row first. It is not
+sorted, and nothing about a label's type changes that: an axis of strings, of
+integers and of timestamps are all read in the order they arrive. The order is
+observable — [`shift`](operators.md#shift), `sum_back` and `position()` all walk
+it — so a consumer that sorted would answer `shift(p, over=snapshot, offset=1)`
+with a different row, and the file could not tell you which it meant. A model
+wanting a particular order states it in the source it hands over.
+
+**One row per coordinate.** A parameter's table carries each coordinate of its
+`dims` at most once, and a second row for one coordinate is an error naming the
+coordinate — never a last-wins, a first-wins or a sum, each of which is a
+defensible reading, which is exactly why the file may not leave the choice
+open. A lookup's map obeys the same rule one axis over, and says so under
+`lookups` below: it is single-valued per label of `over`.
+
+_At most_ once, rather than exactly once: a coordinate with no row is
+[absence](absence.md), which is how a model masks — and whether a given
+parameter meant to mask is
+[`coverage`](declarations.md), which it declares rather than leaving to be
+inferred from the table.
 
 ## `lookups`
 

docs/reference/language/reading.md

@@ -139,3 +139,35 @@ The footprint stops at the kind. A sink that takes a window but not a wrapped
 one reads `Window in footprint.shapes` and then walks: `wrap`, `partition` and
 a named width are refinements without end, and each is one line once the set
 has said where to look.
+
+## Fixing a decision somebody else made
+
+A myopic pathway, a rolling horizon and a Benders subproblem share one move: a
+variable stops being a decision and becomes a number somebody else chose.
+
+```python
+from math_spec import fix
+
+sorted(fix(spec, 'cost').parameters)  # ['bp_x', 'bp_y', 'cost']
+```
+
+A myopic step fixes what earlier periods built, which is many at once, so
+`fix` takes every name in one call and validates once at the end of it. The
+name does not move, so every expression naming it goes on reading and the
+subproblem is a call rather than a second file to keep in step. What it is not
+is every half of a decomposition: a Benders _master_ has the dispatch **gone**
+rather than fixed, and fixing every variable a constraint names leaves a row
+that decides nothing, which the language refuses.
+
+Two of the translations are decisions rather than copies, and are why this is
+here rather than four lines in a driver. A variable masked by `where:` has rows
+that do not exist, so as a parameter it is
+[`coverage: masked`](declarations.md#parameters) — the obvious rewrite leaves it
+`total`, which claims a number everywhere and binds cleanly against data that
+has none. And a `binary` or `integer` variable becomes an `int` parameter,
+never a `float` one, because the values are whole and a `bool` would be a mask
+rather than something a constraint multiplies by.
+
+Bounds are dropped, which is the one thing lost: they constrained a decision
+the model no longer makes, and whether the supplied numbers respect them is a
+question about data.

examples/pypsa.yaml

@@ -67,9 +67,11 @@ parameters:
   Generator_ramp_limit_up:
     description: most a generator may raise its output between snapshots, per unit of nominal power; no value means no limit
     dims: [generator]
+    coverage: masked
   Generator_ramp_limit_down:
     description: most a generator may lower its output between snapshots, per unit of nominal power; no value means no limit
     dims: [generator]
+    coverage: masked
   Generator_ramp_limit_start_up:
     description: most output in the snapshot a unit starts, per unit of nominal power
     dims: [generator]
@@ -107,6 +109,7 @@ parameters:
   Generator_p_nom_mod:
     description: the module size a build comes in whole numbers of; no value means the build is continuous
     dims: [generator]
+    coverage: masked
   Generator_modules_installed:
     description: >-
       how many whole modules a committable build has in place: `Generator_p_nom
@@ -126,9 +129,11 @@ parameters:
   Link_ramp_limit_up:
     description: most a link may raise its flow between snapshots, per unit of nominal power; no value means no limit
     dims: [link]
+    coverage: masked
   Link_ramp_limit_down:
     description: most a link may lower its flow between snapshots, per unit of nominal power; no value means no limit
     dims: [link]
+    coverage: masked
   Link_p_nom:
     description: nominal power
     dims: [link]

examples/pypsa_linearized_uc.yaml

@@ -72,9 +72,11 @@ parameters:
   Generator_ramp_limit_up:
     description: most a generator may raise its output between snapshots, per unit of nominal power; no value means no limit
     dims: [generator]
+    coverage: masked
   Generator_ramp_limit_down:
     description: most a generator may lower its output between snapshots, per unit of nominal power; no value means no limit
     dims: [generator]
+    coverage: masked
   Generator_ramp_limit_start_up:
     description: most output in the snapshot a unit starts, per unit of nominal power
     dims: [generator]

examples/pypsa_multi_period.yaml

@@ -55,6 +55,7 @@ parameters:
   Carrier_max_growth:
     description: most capacity of a carrier that may be added in a period; no value means no limit
     dims: [carrier]
+    coverage: masked
   Carrier_max_relative_growth:
     description: share of the previous period's additions that may be added on top
     dims: [carrier]

schema/math-spec.schema.json

@@ -356,6 +356,15 @@
       "additionalProperties": false,
       "description": "A declared parameter with dims and dtype.",
       "properties": {
+        "coverage": {
+          "default": "total",
+          "enum": [
+            "total",
+            "masked"
+          ],
+          "title": "Coverage",
+          "type": "string"
+        },
         "description": {
           "anyOf": [
             {

src/math_spec/__init__.py

@@ -42,6 +42,7 @@
     edge_error,
     unknown_operator_message,
 )
+from math_spec.transforms import fix
 
 # Last: `math_spec.typesetting` reaches back for the two conversions, so those
 # must be bound before it is imported.
@@ -78,6 +79,7 @@
     'call_shape_error',
     'did_you_mean',
     'edge_error',
+    'fix',
     'program',
     'schema_error',
     'to_latex',

src/math_spec/lowering.py

@@ -56,7 +56,7 @@
 from math_spec.where_parser import AndNode, NotNode, WhereNode
 
 if TYPE_CHECKING:
-    from collections.abc import Callable
+    from collections.abc import Callable, Mapping
     from pathlib import Path
     from typing import Any
 
@@ -141,7 +141,7 @@ def lower_program(schema: _ExpandedSpec) -> program.Program:
         for name, how in derivations_of(block, ex).items()
     }
     parameters = {
-        name: program.ParameterDeclaration(tuple(pdef.dims), pdef.dtype, derivations.get(name))
+        name: program.ParameterDeclaration(tuple(pdef.dims), pdef.dtype, derivations.get(name), pdef.coverage)
         for name, pdef in expanded.parameters.items()
     }
 
@@ -212,6 +212,7 @@ def lower_program(schema: _ExpandedSpec) -> program.Program:
         for sname, sdef in expanded.sos.items()
     }
     expressions = {name: _lower_expression(expanded, ns, name) for name in expanded.expressions}
+    _refuse_a_mask_where_absence_has_no_reading(parameters, variables, constraints, objective, expressions)
     return program.Program(
         parameters=parameters,
         variables=variables,
@@ -441,3 +442,48 @@ def _bound_expression(value: float | str) -> program.ExpressionNode:
     if isinstance(value, str):
         return program.Parameter(value)
     return program.Constant(value)
+
+
+#: Where a missing value has no reading that contributes nothing, and so is
+#: refused rather than filled — the two positions named in rule 8.
+_NO_READING_FOR_ABSENCE = 'a bound', 'a divisor'
+
+
+def _refuse_a_mask_where_absence_has_no_reading(
+    parameters: Mapping[str, program.ParameterDeclaration],
+    variables: Mapping[str, program.VariableDeclaration],
+    constraints: Mapping[str, program.ConstraintDeclaration],
+    objective: program.ObjectiveDeclaration | None,
+    expressions: Mapping[str, program.ExpressionNode],
+) -> None:
+    """Refuse ``coverage: masked`` in the two positions rule 8 gives absence no reading.
+
+    A bound and a divisor are the positions where a missing value cannot read
+    as "contributes nothing": an absent bound is no bound rather than an open
+    one, and an absent divisor is no quotient at all. A parameter declaring
+    itself a mask therefore cannot stand in either, and the file says so before
+    any data arrives — where the same fault would otherwise surface as a bind
+    error against whichever rows the data happened to carry.
+
+    Both positions are read off the lowered declarations rather than the file,
+    so a parameter reaching one through a macro or a named expression is caught
+    on the same footing as one written there directly.
+    """
+    every = (
+        *(node for vdef in variables.values() for node in (vdef.lower, vdef.upper)),
+        *(node for cdef in constraints.values() for node in (cdef.lhs, cdef.rhs)),
+        *((objective.expression,) if objective is not None else ()),
+        *expressions.values(),
+    )
+    bounded = program.parameters_of(*(node for vdef in variables.values() for node in (vdef.lower, vdef.upper)))
+    positions = dict.fromkeys(bounded, 'a bound')
+    positions |= dict.fromkeys(program.divisor_parameters(*every), 'a divisor')
+    for name, position in sorted(positions.items()):
+        if parameters[name].coverage == 'masked':
+            raise LanguageError(
+                f"parameter '{name}' is declared `coverage: masked`, and stands as {position}. "
+                f'A missing row is absence, and absence has no reading there — an absent bound is '
+                f'no bound rather than an open one, and an absent divisor is no quotient at all. '
+                f"Declare `coverage: total` on '{name}' where its table does carry every "
+                f'coordinate, or move the mask onto the declaration that wants it, as a `where:`.'
+            )

src/math_spec/model.py

@@ -88,6 +88,13 @@ def _reject_unknown_keys(cls, data: Any) -> Any:
 #: indexes by.
 ParameterDtype = Literal['float', 'int', 'bool', 'str']
 
+#: What a parameter's table is required to carry. ``total`` is every coordinate
+#: its ``dims`` reach; ``masked`` says a missing row is deliberate — the
+#: parameter *is* a mask, and a missing row reads as the identity of the
+#: position it stands in. The two are indistinguishable in the data, which is
+#: why the declaration says which was meant rather than a consumer guessing.
+ParameterCoverage = Literal['total', 'masked']
+
 #: The domain a variable may declare.
 VariableDomain = Literal['continuous', 'integer', 'binary']
 
@@ -120,6 +127,7 @@ def _reject_unknown_keys(cls, data: Any) -> Any:
 #: The set form of each vocabulary above, for callers that want membership.
 DIMENSION_DTYPES = frozenset(get_args(DimensionDtype))
 PARAMETER_DTYPES = frozenset(get_args(ParameterDtype))
+PARAMETER_COVERAGE = frozenset(get_args(ParameterCoverage))
 #: The parameter dtypes that stand where a number belongs — a coefficient, a
 #: term, a divisor, a bound. A label selects and a flag masks; neither is one.
 NUMERIC_DTYPES = frozenset({'float', 'int'})
@@ -202,6 +210,7 @@ class ParameterBlock(_StrictBlock):
 
     dims: list[str]
     dtype: ParameterDtype = 'float'
+    coverage: ParameterCoverage = 'total'
     description: str | None = None
 
 

src/math_spec/program.py

@@ -108,6 +108,7 @@
     'ObjectiveDeclaration',
     'ObjectiveSense',
     'Parameter',
+    'ParameterCoverage',
     'ParameterDeclaration',
     'ParameterDtype',
     'PiecewiseDeclaration',
@@ -168,6 +169,10 @@
 #: What a parameter's values are (:data:`~math_spec.model.ParameterDtype`).
 ParameterDtype = _model.ParameterDtype
 
+#: Whether a parameter's table must carry every coordinate of its dims
+#: (:data:`~math_spec.model.ParameterCoverage`).
+ParameterCoverage = _model.ParameterCoverage
+
 #: What a masked variable's non-existence means
 #: (:data:`~math_spec.model.VariableAbsence`).
 VariableAbsence = _model.VariableAbsence
@@ -720,6 +725,10 @@ class ParameterDeclaration:
     #: follows: the caller binds a declared parameter, and an emitted one is
     #: built from the block's own breakpoints the way its derivation says.
     derivation: Derivation | None = None
+    #: Whether the table must carry every coordinate of *dims*. ``masked`` says
+    #: a missing row is absence the model means, so the declaration rather than
+    #: the data decides how a short table reads.
+    coverage: ParameterCoverage = 'total'
 
 
 @dataclass(frozen=True)