From 33b7db70e267f6ef1268e6931847f07aff77826d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 13:39:40 +0000 Subject: [PATCH 1/3] docs(language): the rules binding obeys belong to the language, not to whichever engine reads the data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three facts decide which model a file and a table make together — where a dimension's members come from, the order they stand in, and that a coordinate appears at most once. Each was real and each was written down in a consumer, so a second consumer would have inherited none of them. Co-Authored-By: Claude (cherry picked from commit 1e2b8a951dda3391005735e983ffdfa7d4f1f9db) --- docs/reference/language/declarations.md | 5 +++- docs/reference/language/dimensions.md | 36 +++++++++++++++++++++++-- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/docs/reference/language/declarations.md b/docs/reference/language/declarations.md index caca3045..ad89f963 100644 --- a/docs/reference/language/declarations.md +++ b/docs/reference/language/declarations.md @@ -20,7 +20,10 @@ characters rather than as a symbol. Write the thing rather than its symbol — ## `parameters` Declared shape only; the numbers bind by name at run time, in whatever -consumes the AST. +consumes the AST. What that binding may not decide for itself — where a +dimension's members come from, the order they stand in, and that a table +carries each coordinate at most once — is in +[dimensions](dimensions.md). ```yaml dimensions: diff --git a/docs/reference/language/dimensions.md b/docs/reference/language/dimensions.md index 8313c310..106bd687 100644 --- a/docs/reference/language/dimensions.md +++ b/docs/reference/language/dimensions.md @@ -32,8 +32,40 @@ in step by hand. **One master coordinate set per dimension, resolved before any data binds.** Every parameter is reindexed onto it, so two tables that disagree about which snapshots exist is an error at load time rather than a silently truncated -model. Where the coordinates come from, and in what order, is settled when -data is bound — which this package declares the shape of and does not do. +model. Which coordinates those are, and in what order they stand, is data's to +say — and the three rules below say how it says it. + +### Binding is the language's, even though the data is not + +The file declares an axis; the data supplies its members. Between those two +sentences sit three facts that decide **which model a file and a table make +together** — and a consumer answering any of them differently would build a +different model from the same two inputs. So they are the language's, and a +consumer implements them rather than choosing them. + +**The dimension's own source supplies its members.** They are read from the key +named after the dimension, and from nothing else: a parameter's table is read +for values, never for labels, and a lookup's map is not a claim about which +members exist. A dimension nothing supplies is an error naming it, not an empty +axis — an axis with no members would delete every row indexed by it, silently. + +**Their order is the order that source gives them**, first row first. It is not +sorted, and nothing about a label's type changes that: an axis of strings, of +integers and of timestamps are all read in the order they arrive. The order is +observable — [`shift`](operators.md#shift), `sum_back` and `position()` all walk +it — so a consumer that sorted would answer `shift(p, over=snapshot, offset=1)` +with a different row, and the file could not tell you which it meant. A model +wanting a particular order states it in the source it hands over. + +**One row per coordinate.** A parameter's table carries each coordinate of its +`dims` at most once, and a second row for one coordinate is an error naming the +coordinate — never a last-wins, a first-wins or a sum, each of which is a +defensible reading, which is exactly why the file may not leave the choice +open. A lookup's map obeys the same rule one axis over, and says so under +`lookups` below: it is single-valued per label of `over`. + +_At most_ once, rather than exactly once: a coordinate with no row is +[absence](absence.md), which is how a model masks. ## `lookups` From 981aa956c83c974d77732575177bb207dd9a97aa Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 13:45:45 +0000 Subject: [PATCH 2/3] feat(language): a parameter says whether its table must cover every coordinate, so a row lost in preparation is not read as a mask MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A table short of a coordinate and a table that never had one are identical in the data and mean opposite things. `coverage:` says which was meant, and a parameter declaring itself a mask is refused as a bound or a divisor — the two positions where absence has no reading — before any data arrives. Co-Authored-By: Claude (cherry picked from commit 9a8045cb2afa55b4778536aa6194e8244ee4c193) --- docs/reference/language/declarations.md | 22 ++++++++ docs/reference/language/dimensions.md | 5 +- examples/pypsa.yaml | 5 ++ examples/pypsa_linearized_uc.yaml | 2 + examples/pypsa_multi_period.yaml | 1 + schema/math-spec.schema.json | 9 +++ src/math_spec/lowering.py | 50 ++++++++++++++++- src/math_spec/model.py | 9 +++ src/math_spec/program.py | 9 +++ tests/test_lowering.py | 11 +++- tests/test_validation.py | 73 +++++++++++++++++++++++++ 11 files changed, 192 insertions(+), 4 deletions(-) diff --git a/docs/reference/language/declarations.md b/docs/reference/language/declarations.md index ad89f963..d16929f3 100644 --- a/docs/reference/language/declarations.md +++ b/docs/reference/language/declarations.md @@ -39,8 +39,30 @@ parameters: | ------------- | ------------------------------------------------------------ | --------------- | | `dims` | required — the dimensions it is indexed by; `[]` is a scalar | | | `dtype` | `float`, `int`, `bool`, `str` | default `float` | +| `coverage` | `total`, `masked` | default `total` | | `description` | free text | default `null` | +**`coverage` says whether a missing row was meant.** A table short of a +coordinate and a table that never had one look identical in the data, and they +mean opposite things: `total` is the claim that every coordinate the `dims` +reach has a value, so a row that went missing in preparation is an error rather +than a mask; `masked` is the file saying the gap is the point — the parameter +_is_ a mask, and a coordinate it leaves out is [absence](absence.md). + +```yaml +parameters: + cost: { dims: [generator] } # total: every generator has one + ramp_limit: { dims: [generator], coverage: masked } # no row means no limit +``` + +Without it the reading is a consumer's to pick, and two consumers picking +differently would build different models from one file and one table — so the +declaration says it and no consumer guesses. **The default is `total`** because +that is what the other rules already assume: a bound and a divisor +[refuse absence outright](absence.md), so a parameter reaching either must +cover its rows, and a file declaring `masked` in those positions is a load +error naming the rewrite. + **`dtype` is a claim about the values, and the column has to be it.** It decides four things — whether the name is a value in an [expression](expressions.md) at all, what a `where` comparison is checked diff --git a/docs/reference/language/dimensions.md b/docs/reference/language/dimensions.md index 106bd687..9b64c70e 100644 --- a/docs/reference/language/dimensions.md +++ b/docs/reference/language/dimensions.md @@ -65,7 +65,10 @@ open. A lookup's map obeys the same rule one axis over, and says so under `lookups` below: it is single-valued per label of `over`. _At most_ once, rather than exactly once: a coordinate with no row is -[absence](absence.md), which is how a model masks. +[absence](absence.md), which is how a model masks — and whether a given +parameter meant to mask is +[`coverage`](declarations.md), which it declares rather than leaving to be +inferred from the table. ## `lookups` diff --git a/examples/pypsa.yaml b/examples/pypsa.yaml index 006372e0..e4aaa1e2 100644 --- a/examples/pypsa.yaml +++ b/examples/pypsa.yaml @@ -67,9 +67,11 @@ parameters: Generator_ramp_limit_up: description: most a generator may raise its output between snapshots, per unit of nominal power; no value means no limit dims: [generator] + coverage: masked Generator_ramp_limit_down: description: most a generator may lower its output between snapshots, per unit of nominal power; no value means no limit dims: [generator] + coverage: masked Generator_ramp_limit_start_up: description: most output in the snapshot a unit starts, per unit of nominal power dims: [generator] @@ -107,6 +109,7 @@ parameters: Generator_p_nom_mod: description: the module size a build comes in whole numbers of; no value means the build is continuous dims: [generator] + coverage: masked Generator_modules_installed: description: >- how many whole modules a committable build has in place: `Generator_p_nom @@ -126,9 +129,11 @@ parameters: Link_ramp_limit_up: description: most a link may raise its flow between snapshots, per unit of nominal power; no value means no limit dims: [link] + coverage: masked Link_ramp_limit_down: description: most a link may lower its flow between snapshots, per unit of nominal power; no value means no limit dims: [link] + coverage: masked Link_p_nom: description: nominal power dims: [link] diff --git a/examples/pypsa_linearized_uc.yaml b/examples/pypsa_linearized_uc.yaml index 5c300c0a..1596174d 100644 --- a/examples/pypsa_linearized_uc.yaml +++ b/examples/pypsa_linearized_uc.yaml @@ -72,9 +72,11 @@ parameters: Generator_ramp_limit_up: description: most a generator may raise its output between snapshots, per unit of nominal power; no value means no limit dims: [generator] + coverage: masked Generator_ramp_limit_down: description: most a generator may lower its output between snapshots, per unit of nominal power; no value means no limit dims: [generator] + coverage: masked Generator_ramp_limit_start_up: description: most output in the snapshot a unit starts, per unit of nominal power dims: [generator] diff --git a/examples/pypsa_multi_period.yaml b/examples/pypsa_multi_period.yaml index adcd1e4f..abb0e796 100644 --- a/examples/pypsa_multi_period.yaml +++ b/examples/pypsa_multi_period.yaml @@ -55,6 +55,7 @@ parameters: Carrier_max_growth: description: most capacity of a carrier that may be added in a period; no value means no limit dims: [carrier] + coverage: masked Carrier_max_relative_growth: description: share of the previous period's additions that may be added on top dims: [carrier] diff --git a/schema/math-spec.schema.json b/schema/math-spec.schema.json index 2666a975..fbcda6ab 100644 --- a/schema/math-spec.schema.json +++ b/schema/math-spec.schema.json @@ -356,6 +356,15 @@ "additionalProperties": false, "description": "A declared parameter with dims and dtype.", "properties": { + "coverage": { + "default": "total", + "enum": [ + "total", + "masked" + ], + "title": "Coverage", + "type": "string" + }, "description": { "anyOf": [ { diff --git a/src/math_spec/lowering.py b/src/math_spec/lowering.py index f51b3097..46052749 100644 --- a/src/math_spec/lowering.py +++ b/src/math_spec/lowering.py @@ -56,7 +56,7 @@ from math_spec.where_parser import AndNode, NotNode, WhereNode if TYPE_CHECKING: - from collections.abc import Callable + from collections.abc import Callable, Mapping from pathlib import Path from typing import Any @@ -141,7 +141,7 @@ def lower_program(schema: _ExpandedSpec) -> program.Program: for name, how in derivations_of(block, ex).items() } parameters = { - name: program.ParameterDeclaration(tuple(pdef.dims), pdef.dtype, derivations.get(name)) + name: program.ParameterDeclaration(tuple(pdef.dims), pdef.dtype, derivations.get(name), pdef.coverage) for name, pdef in expanded.parameters.items() } @@ -212,6 +212,7 @@ def lower_program(schema: _ExpandedSpec) -> program.Program: for sname, sdef in expanded.sos.items() } expressions = {name: _lower_expression(expanded, ns, name) for name in expanded.expressions} + _refuse_a_mask_where_absence_has_no_reading(parameters, variables, constraints, objective, expressions) return program.Program( parameters=parameters, variables=variables, @@ -441,3 +442,48 @@ def _bound_expression(value: float | str) -> program.ExpressionNode: if isinstance(value, str): return program.Parameter(value) return program.Constant(value) + + +#: Where a missing value has no reading that contributes nothing, and so is +#: refused rather than filled — the two positions named in rule 8. +_NO_READING_FOR_ABSENCE = 'a bound', 'a divisor' + + +def _refuse_a_mask_where_absence_has_no_reading( + parameters: Mapping[str, program.ParameterDeclaration], + variables: Mapping[str, program.VariableDeclaration], + constraints: Mapping[str, program.ConstraintDeclaration], + objective: program.ObjectiveDeclaration | None, + expressions: Mapping[str, program.ExpressionNode], +) -> None: + """Refuse ``coverage: masked`` in the two positions rule 8 gives absence no reading. + + A bound and a divisor are the positions where a missing value cannot read + as "contributes nothing": an absent bound is no bound rather than an open + one, and an absent divisor is no quotient at all. A parameter declaring + itself a mask therefore cannot stand in either, and the file says so before + any data arrives — where the same fault would otherwise surface as a bind + error against whichever rows the data happened to carry. + + Both positions are read off the lowered declarations rather than the file, + so a parameter reaching one through a macro or a named expression is caught + on the same footing as one written there directly. + """ + every = ( + *(node for vdef in variables.values() for node in (vdef.lower, vdef.upper)), + *(node for cdef in constraints.values() for node in (cdef.lhs, cdef.rhs)), + *((objective.expression,) if objective is not None else ()), + *expressions.values(), + ) + bounded = program.parameters_of(*(node for vdef in variables.values() for node in (vdef.lower, vdef.upper))) + positions = dict.fromkeys(bounded, 'a bound') + positions |= dict.fromkeys(program.divisor_parameters(*every), 'a divisor') + for name, position in sorted(positions.items()): + if parameters[name].coverage == 'masked': + raise LanguageError( + f"parameter '{name}' is declared `coverage: masked`, and stands as {position}. " + f'A missing row is absence, and absence has no reading there — an absent bound is ' + f'no bound rather than an open one, and an absent divisor is no quotient at all. ' + f"Declare `coverage: total` on '{name}' where its table does carry every " + f'coordinate, or move the mask onto the declaration that wants it, as a `where:`.' + ) diff --git a/src/math_spec/model.py b/src/math_spec/model.py index 5f10d978..83da46b1 100644 --- a/src/math_spec/model.py +++ b/src/math_spec/model.py @@ -88,6 +88,13 @@ def _reject_unknown_keys(cls, data: Any) -> Any: #: indexes by. ParameterDtype = Literal['float', 'int', 'bool', 'str'] +#: What a parameter's table is required to carry. ``total`` is every coordinate +#: its ``dims`` reach; ``masked`` says a missing row is deliberate — the +#: parameter *is* a mask, and a missing row reads as the identity of the +#: position it stands in. The two are indistinguishable in the data, which is +#: why the declaration says which was meant rather than a consumer guessing. +ParameterCoverage = Literal['total', 'masked'] + #: The domain a variable may declare. VariableDomain = Literal['continuous', 'integer', 'binary'] @@ -120,6 +127,7 @@ def _reject_unknown_keys(cls, data: Any) -> Any: #: The set form of each vocabulary above, for callers that want membership. DIMENSION_DTYPES = frozenset(get_args(DimensionDtype)) PARAMETER_DTYPES = frozenset(get_args(ParameterDtype)) +PARAMETER_COVERAGE = frozenset(get_args(ParameterCoverage)) #: The parameter dtypes that stand where a number belongs — a coefficient, a #: term, a divisor, a bound. A label selects and a flag masks; neither is one. NUMERIC_DTYPES = frozenset({'float', 'int'}) @@ -202,6 +210,7 @@ class ParameterBlock(_StrictBlock): dims: list[str] dtype: ParameterDtype = 'float' + coverage: ParameterCoverage = 'total' description: str | None = None diff --git a/src/math_spec/program.py b/src/math_spec/program.py index 2399d5ef..0698c529 100644 --- a/src/math_spec/program.py +++ b/src/math_spec/program.py @@ -108,6 +108,7 @@ 'ObjectiveDeclaration', 'ObjectiveSense', 'Parameter', + 'ParameterCoverage', 'ParameterDeclaration', 'ParameterDtype', 'PiecewiseDeclaration', @@ -168,6 +169,10 @@ #: What a parameter's values are (:data:`~math_spec.model.ParameterDtype`). ParameterDtype = _model.ParameterDtype +#: Whether a parameter's table must carry every coordinate of its dims +#: (:data:`~math_spec.model.ParameterCoverage`). +ParameterCoverage = _model.ParameterCoverage + #: What a masked variable's non-existence means #: (:data:`~math_spec.model.VariableAbsence`). VariableAbsence = _model.VariableAbsence @@ -720,6 +725,10 @@ class ParameterDeclaration: #: follows: the caller binds a declared parameter, and an emitted one is #: built from the block's own breakpoints the way its derivation says. derivation: Derivation | None = None + #: Whether the table must carry every coordinate of *dims*. ``masked`` says + #: a missing row is absence the model means, so the declaration rather than + #: the data decides how a short table reads. + coverage: ParameterCoverage = 'total' @dataclass(frozen=True) diff --git a/tests/test_lowering.py b/tests/test_lowering.py index e9fda9ac..ba972044 100644 --- a/tests/test_lowering.py +++ b/tests/test_lowering.py @@ -27,7 +27,7 @@ from math_spec import LanguageError, Spec from math_spec.exclusivity import overlapping from math_spec.expression_parser import FunctionCallNode, NumberNode -from math_spec.lowering import _Lowering, lower_program +from math_spec.lowering import _Lowering, lower_program, to_program from math_spec.piecewise import expand_piecewise from math_spec.program import ( QUADRATIC_POSITIONS, @@ -758,3 +758,12 @@ def test_a_cased_expression_is_readable_by_the_name_the_file_wrote(): assert isinstance(program.named_expressions['previous'], program_module.Cases), ( 'a cased expression reaches the program as the node, not as its fallback arm alone' ) + + +def test_a_parameter_covers_every_coordinate_unless_it_says_otherwise(): + """The default is the claim rule 8 already makes of a bound and a divisor: + a table carries what its dims reach. ``masked`` is the file saying the + missing row was meant.""" + program = to_program(override(SMALL_MODEL, **{'parameters.k.coverage': 'masked'})) + assert program.parameters['c'].coverage == 'total', 'a parameter that says nothing covers its dims' + assert program.parameters['k'].coverage == 'masked', 'and one that says so is carried through unchanged' diff --git a/tests/test_validation.py b/tests/test_validation.py index 130ea7af..54df5a8e 100644 --- a/tests/test_validation.py +++ b/tests/test_validation.py @@ -14,6 +14,7 @@ from math_spec._yaml import parse_yaml from math_spec.errors import DimensionError, LanguageError, SchemaError +from math_spec.lowering import to_program from math_spec.resolution import Namespace, where_of from math_spec.validation import to_spec from math_spec.where_parser import DimensionPositionNode @@ -848,3 +849,75 @@ def test_a_boolean_is_still_not_an_expression(self): """`true` is not arithmetic, and an error naming the type reads better than one naming `'True'`.""" with pytest.raises(SchemaError, match='valid string'): _schema(**{'expressions.always': {'expression': True}}) + + +class TestParameterCoverage: + """``coverage: masked`` is a claim, and rule 8 gives it two places it cannot stand. + + A bound and a divisor are where a missing value has no reading that + contributes nothing, so a parameter calling itself a mask is refused there + before any data arrives — rather than binding and failing against whichever + rows the caller's table happened to carry. + """ + + @pytest.mark.parametrize( + ('patch', 'position'), + [ + pytest.param( + {'parameters.c.coverage': 'masked', 'variables.p.bounds': {'upper': 'c'}}, + 'a bound', + id='masked-as-an-upper-bound', + ), + pytest.param( + {'parameters.c.coverage': 'masked', 'variables.p.bounds': {'lower': 'c'}}, + 'a bound', + id='masked-as-a-lower-bound', + ), + pytest.param( + { + 'parameters.c.coverage': 'masked', + 'constraints': {'cap': {'foreach': ['g'], 'expression': 'p / c <= 1'}}, + }, + 'a divisor', + id='masked-as-a-divisor', + ), + pytest.param( + { + 'parameters.c.coverage': 'masked', + 'expressions': {'scaled': 'p / c'}, + 'constraints': {'cap': {'foreach': ['g'], 'expression': 'scaled <= 1'}}, + }, + 'a divisor', + id='masked-as-a-divisor-reached-through-a-named-expression', + ), + ], + ) + def test_a_masked_parameter_is_refused_where_absence_has_no_reading(self, patch, position): + with pytest.raises(LanguageError) as exc: + to_program(override(SMALL_MODEL, **patch)) + assert "parameter 'c'" in str(exc.value), 'the message names the parameter that has to change' + assert position in str(exc.value), f'the message names the position, which decides the rewrite: {position}' + assert 'coverage: total' in str(exc.value), 'the message names the rewrite, not only the fault' + + @pytest.mark.parametrize( + ('patch'), + [ + pytest.param( + {'constraints': {'cap': {'foreach': ['g'], 'expression': 'p * c <= 1'}}}, + id='a-coefficient-reads-as-zero', + ), + pytest.param( + {'variables.p.where': 'c > 0'}, + id='a-where-reads-as-false', + ), + pytest.param( + {'constraints': {'cap': {'foreach': ['g'], 'expression': 'p + c <= 1'}}}, + id='a-term-reads-as-zero', + ), + ], + ) + def test_a_masked_parameter_stands_wherever_absence_does_have_a_reading(self, patch): + """The guard is the two positions of rule 8 and not a third: a mask is + what a sparse table is *for*, so refusing it as a coefficient, a term or + a where would refuse the construct the declaration exists to describe.""" + to_program(override(SMALL_MODEL, **{'parameters.c.coverage': 'masked', **patch})) From 3348d7003a499cc5bbb780d92d1072e6a3f0ea29 Mon Sep 17 00:00:00 2001 From: FBumann <117816358+FBumann@users.noreply.github.com> Date: Mon, 31 Aug 2026 20:27:19 +0200 Subject: [PATCH 3/3] feat(language): a decided variable becomes a supplied number, so a subproblem is a call rather than a second file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A myopic pathway, a rolling horizon and a Benders subproblem share one move: a variable stops being a decision and becomes a number somebody else chose. fix takes every name in one call and validates once at the end. Two translations are decisions, not copies: a where-masked variable becomes coverage: masked, and a binary or integer variable becomes an int parameter — never a float, never a bool that would read as a mask. Rebased onto the parameter-coverage branch (#243) it needs for Coverage, off the closed #247/#248 stack it was authored on; only the fix additions are kept. Co-Authored-By: Claude --- docs/reference/language/reading.md | 32 +++++++++ src/math_spec/__init__.py | 2 + src/math_spec/transforms.py | 101 ++++++++++++++++++++++++++ tests/test_public_surface.py | 2 + tests/test_reading_page.py | 2 +- tests/test_transforms.py | 109 +++++++++++++++++++++++++++++ 6 files changed, 247 insertions(+), 1 deletion(-) create mode 100644 src/math_spec/transforms.py create mode 100644 tests/test_transforms.py diff --git a/docs/reference/language/reading.md b/docs/reference/language/reading.md index 6bcca9d1..bd224aaa 100644 --- a/docs/reference/language/reading.md +++ b/docs/reference/language/reading.md @@ -139,3 +139,35 @@ The footprint stops at the kind. A sink that takes a window but not a wrapped one reads `Window in footprint.shapes` and then walks: `wrap`, `partition` and a named width are refinements without end, and each is one line once the set has said where to look. + +## Fixing a decision somebody else made + +A myopic pathway, a rolling horizon and a Benders subproblem share one move: a +variable stops being a decision and becomes a number somebody else chose. + +```python +from math_spec import fix + +sorted(fix(spec, 'cost').parameters) # ['bp_x', 'bp_y', 'cost'] +``` + +A myopic step fixes what earlier periods built, which is many at once, so +`fix` takes every name in one call and validates once at the end of it. The +name does not move, so every expression naming it goes on reading and the +subproblem is a call rather than a second file to keep in step. What it is not +is every half of a decomposition: a Benders _master_ has the dispatch **gone** +rather than fixed, and fixing every variable a constraint names leaves a row +that decides nothing, which the language refuses. + +Two of the translations are decisions rather than copies, and are why this is +here rather than four lines in a driver. A variable masked by `where:` has rows +that do not exist, so as a parameter it is +[`coverage: masked`](declarations.md#parameters) — the obvious rewrite leaves it +`total`, which claims a number everywhere and binds cleanly against data that +has none. And a `binary` or `integer` variable becomes an `int` parameter, +never a `float` one, because the values are whole and a `bool` would be a mask +rather than something a constraint multiplies by. + +Bounds are dropped, which is the one thing lost: they constrained a decision +the model no longer makes, and whether the supplied numbers respect them is a +question about data. diff --git a/src/math_spec/__init__.py b/src/math_spec/__init__.py index d1c7eeaa..44956ef6 100644 --- a/src/math_spec/__init__.py +++ b/src/math_spec/__init__.py @@ -42,6 +42,7 @@ edge_error, unknown_operator_message, ) +from math_spec.transforms import fix # Last: `math_spec.typesetting` reaches back for the two conversions, so those # must be bound before it is imported. @@ -78,6 +79,7 @@ 'call_shape_error', 'did_you_mean', 'edge_error', + 'fix', 'program', 'schema_error', 'to_latex', diff --git a/src/math_spec/transforms.py b/src/math_spec/transforms.py new file mode 100644 index 00000000..09e90292 --- /dev/null +++ b/src/math_spec/transforms.py @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +"""Rewriting a model into the one a decomposition solves. + +A myopic pathway, a rolling horizon and a Benders subproblem share one move: +**a variable stops being a decision and becomes a number somebody else chose.** +The subproblem is the same model at a capacity someone picked; the myopic step +is the same model with what earlier periods already built. Written by hand that +is a second file to keep in step with the first, and the drift between them is +a bug nothing catches. + +It is not every half of a decomposition. A Benders *master* is not the model +with the dispatch fixed — it is the model with the dispatch **gone**, which is +a different move and not this one. Fixing every variable a constraint names +leaves a row that decides nothing, and the language says so. + +It reads like a rename and it is not, which is why it is here rather than four +lines in every driver. A fixed variable's *mask* becomes a claim about its +data, and its *domain* becomes a dtype — two decisions a driver would each make +differently, and one of them silently: + +- A variable masked by ``where:`` has rows that do not exist. As a parameter + those rows have no value, so it is ``coverage: masked``. The obvious + rewrite leaves it ``total``, which claims every coordinate has a number and + is the wrong answer that binds cleanly. +- A ``binary`` or ``integer`` variable becomes an ``int`` parameter, never a + ``float`` one: the values are whole and a consumer reading the declaration + is entitled to know it. + +Bounds are dropped, and that is the one thing lost. They constrained a decision +this model no longer makes; whether the numbers supplied respect them is a +question about data, which this package does not have. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +from math_spec.errors import did_you_mean +from math_spec.validation import to_spec + +if TYPE_CHECKING: + from pathlib import Path + + from math_spec.model import Coverage, ParameterDtype, Spec + +#: What a fixed variable's values are, by the domain it decided over. ``binary`` +#: is ``int`` rather than ``bool``: a flag masks and a number multiplies, and a +#: fixed commitment is multiplied by. +_DTYPE_OF_DOMAIN: dict[str, ParameterDtype] = {'continuous': 'float', 'integer': 'int', 'binary': 'int'} + + +def fix(model: str | Path | dict[str, Any] | Spec, *names: str) -> Spec: + """*model* with each variable in *names* turned into a parameter of the same name. + + Every expression naming one goes on reading, because the name does not + move — which is what makes the rewrite mechanical. A myopic step fixes + what earlier periods built, which is many at once, so the whole set is + named in one call and validated once at the end of it. + + Args: + model: Whatever every other verb takes. + names: Variables the model declares. Naming none is the model itself. + + Returns: + The model as a :class:`~math_spec.model.Spec`, revalidated — so a fix + that made the model unsayable says so here rather than downstream. + + Raises: + KeyError: One of *names* is not a variable, named with the near miss. + LanguageError: The fixed model is not one the language accepts — a + constraint every variable of which is now a number, a set over one + of them, or a curve linking one. + """ + spec = to_spec(model).to_dict() + variables = spec.get('variables') or {} + for name in names: + if name not in variables: + raise KeyError(f"unknown variable '{name}'. " + did_you_mean(name, list(variables))) + spec.setdefault('parameters', {})[name] = _as_parameter(variables.pop(name)) + return to_spec(spec) + + +def _as_parameter(decided: dict[str, Any]) -> dict[str, Any]: + """The parameter declaration a fixed variable becomes. + + The two translations that are decisions rather than copies are ``where:`` + into ``coverage:`` and ``domain:`` into ``dtype:``; the module docstring + says why each is the one it is. + """ + coverage: Coverage = 'masked' if decided.get('where') is not None else 'total' + declaration: dict[str, Any] = { + 'dims': decided['foreach'], + 'dtype': _DTYPE_OF_DOMAIN[decided.get('domain', 'continuous')], + 'coverage': coverage, + } + if (description := decided.get('description')) is not None: + declaration['description'] = description + return declaration diff --git a/tests/test_public_surface.py b/tests/test_public_surface.py index 039ca765..6ad0caa9 100644 --- a/tests/test_public_surface.py +++ b/tests/test_public_surface.py @@ -28,6 +28,8 @@ 'PiecewiseExpansionError', 'did_you_mean', 'schema_error', # the verdicts a consumer asks for rather than re-deriving 'advice', 'Advice', + # a decided variable becoming a supplied number + 'fix', # the closed operator set, and the wording of its refusals 'BUILTIN_NAMES', 'EDGE_WRAP', 'call_shape_error', 'edge_error', 'unknown_operator_message', diff --git a/tests/test_reading_page.py b/tests/test_reading_page.py index f426db44..17909451 100644 --- a/tests/test_reading_page.py +++ b/tests/test_reading_page.py @@ -55,6 +55,6 @@ def test_the_page_shows_the_declarations_the_expansion_emits(tmp_path, monkeypat exec(compile(code, str(PAGE), 'exec'), namespace) claims.extend(_claims(code)) - assert len(claims) == 7, 'every `expression # value` line on the page is checked; one without one is not' + assert len(claims) == 8, 'every `expression # value` line on the page is checked; one without one is not' for expression, claimed in claims: assert eval(expression, namespace) == claimed, f'reading.md says `{expression}` is {claimed}' diff --git a/tests/test_transforms.py b/tests/test_transforms.py new file mode 100644 index 00000000..daf639b8 --- /dev/null +++ b/tests/test_transforms.py @@ -0,0 +1,109 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +"""A decided variable becoming a supplied number, and what that costs it. + +The move itself is a rename and the tests are not about the rename. They are +about the two translations that are decisions — a mask into a coverage claim, +a domain into a dtype — because those are what a driver writing its own four +lines gets wrong, and one of them gets wrong silently. +""" + +from __future__ import annotations + +from typing import Any + +import pytest + +import math_spec as ms +from math_spec import LanguageError +from math_spec.transforms import fix + +MODEL: dict[str, Any] = { + 'dimensions': {'g': {'dtype': 'str'}}, + 'parameters': {'p_max': {'dims': ['g']}, 'cost': {'dims': ['g']}}, + 'variables': { + 'cap': {'foreach': ['g'], 'bounds': {'lower': 0, 'upper': 'p_max'}, 'description': 'capacity built'}, + 'masked': {'foreach': ['g'], 'where': 'p_max > 0', 'bounds': {'lower': 0}}, + 'on': {'foreach': ['g'], 'domain': 'binary'}, + 'count': {'foreach': ['g'], 'domain': 'integer'}, + }, + 'constraints': {'k': {'foreach': ['g'], 'expression': 'cap + masked + on + count <= p_max'}}, + 'objective': {'sense': 'minimize', 'expression': 'sum(cap * cost)'}, +} + + +def test_a_fixed_variable_is_a_parameter_of_the_same_name_and_dims(): + """The name not moving is the whole point: every expression naming it goes + on reading, so a decomposition is this call rather than a second file.""" + spec = fix(MODEL, 'cap') + assert 'cap' not in spec.variables, 'it is no longer decided' + assert spec.parameters['cap'].dims == ['g'], 'and it is supplied over what it was decided over' + assert ms.to_program(spec).constraints['k'], 'the constraint naming it still builds' + + +def test_a_masked_variable_becomes_a_parameter_that_says_its_rows_are_missing(): + """The one that goes wrong silently. A `where:` deleted rows, so as a + parameter those coordinates have no value — and the obvious rewrite leaves + it `total`, claiming a number everywhere and binding cleanly against data + that has none.""" + assert fix(MODEL, 'masked').parameters['masked'].coverage == 'masked', 'the mask became a claim about the data' + assert fix(MODEL, 'cap').parameters['cap'].coverage == 'total', 'and an unmasked one still covers its dims' + + +@pytest.mark.parametrize( + ('name', 'dtype'), + [ + pytest.param('cap', 'float', id='a-continuous-decision-is-a-float'), + pytest.param('count', 'int', id='an-integer-decision-is-an-int'), + pytest.param('on', 'int', id='a-binary-decision-is-an-int-not-a-bool'), + ], +) +def test_a_domain_becomes_the_dtype_its_values_are(name, dtype): + """`bool` is a mask and never a number, so a fixed commitment that a + constraint multiplies by has to arrive as `int`.""" + assert fix(MODEL, name).parameters[name].dtype == dtype, 'the declaration says what the values are' + + +def test_a_description_survives_and_bounds_do_not(): + """A description describes the quantity, which the fix does not change. + Bounds constrained a decision the model no longer makes, and whether the + supplied numbers respect them is a question about data.""" + fixed = fix(MODEL, 'cap').parameters['cap'] + assert fixed.description == 'capacity built', 'still the same quantity' + assert not hasattr(fixed, 'bounds'), 'a parameter has none to carry' + + +def test_one_call_fixes_every_variable_it_names(): + """A myopic step fixes what earlier periods built, which is many at once. + Composing calls does the same, at one revalidation each.""" + spec = fix(MODEL, 'cap', 'count') + assert sorted(spec.variables) == ['masked', 'on'], 'both named variables became parameters' + assert spec.to_dict() == fix(fix(MODEL, 'cap'), 'count').to_dict(), 'and composing the calls says the same' + + +def test_a_constraint_whose_every_variable_is_fixed_is_refused(): + """`fix` revalidates, so a rewrite that made the model unsayable says so + here rather than downstream. A Benders master is this shape and is not this + move: it drops the dispatch rather than fixing it.""" + with pytest.raises(LanguageError) as exc: + fix(MODEL, 'cap', 'masked', 'on', 'count') + assert 'decides nothing' in str(exc.value), 'the language names what is wrong with the rewritten model' + + +def test_an_unknown_variable_is_refused_with_the_near_miss(): + with pytest.raises(KeyError) as exc: + fix(MODEL, 'capp') + assert 'cap' in str(exc.value), 'the message names the variable it was probably reaching for' + + +def test_the_fixed_model_is_still_one_a_reviewer_can_open(tmp_path): + """A decomposition reached by a function rather than a second file is only + an improvement if the thing it reaches is still a file — otherwise it is a + model nobody can review, which is what hard rule 5 refuses.""" + spec = fix(MODEL, 'cap') + written = tmp_path / 'subproblem.yaml' + written.write_text(spec.to_yaml()) + assert ms.to_spec(written).to_dict() == spec.to_dict(), 'the rewritten model round-trips like any other' + assert 'coverage: total' in written.read_text(), 'and the claim about its data is on the page a reviewer reads'