From 5ff14260a09810f641578aad25dd9405a2c90309 Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 16:48:53 +0200 Subject: [PATCH 1/9] feat(language): a file says what it reads under one given key A file declares the columns and row families it reads and does not build under one closed key, given: variables: and given: constraints:. A given name is in the namespace every expression resolves against, has a frame for the dimension check, and dual() may name a given row family. The file typesets on its own, with a Given legend group. The program carries the declarations under Program.given, lowering does not refuse, and advice gains the kind given, one note per declaration a consumer binds. A dimension only a given declaration indexes counts as reached. The schema is regenerated; the golden output does not move. --- docs/about/limits.md | 28 ++- docs/reference/language/declarations.md | 81 ++++++++- docs/reference/language/errors.md | 1 + docs/reference/language/file.md | 5 +- docs/reference/language/index.md | 4 +- docs/reference/reading.md | 32 ++++ schema/math-spec.schema.json | 103 ++++++++++- src/math_spec/advice.py | 37 +++- src/math_spec/boundedness.py | 2 +- src/math_spec/dimensions.py | 4 +- src/math_spec/errors.py | 2 +- src/math_spec/lowering.py | 5 + src/math_spec/model.py | 73 +++++++- src/math_spec/program.py | 40 +++++ src/math_spec/resolution.py | 10 +- src/math_spec/typesetting/symbols.py | 14 +- src/math_spec/typesetting/walk.py | 18 +- tests/test_advice.py | 23 ++- tests/test_given.py | 217 ++++++++++++++++++++++++ tests/test_reading_page.py | 2 +- 20 files changed, 663 insertions(+), 38 deletions(-) create mode 100644 tests/test_given.py diff --git a/docs/about/limits.md b/docs/about/limits.md index a95a0fb3..b20b8ac0 100644 --- a/docs/about/limits.md +++ b/docs/about/limits.md @@ -12,7 +12,7 @@ or keyword. For the rules a model itself has to obey, read ## How a new construct enters -A request for something new is one of three kinds, and the kind decides what it +A request for something new is one of four kinds, and the kind decides what it costs to add. - **A macro** is a template with arguments, written in the file under `macros:`. @@ -26,8 +26,14 @@ costs to add. - **A formulation** is a block that expands into ordinary variables and constraints before the model is built. `piecewise:` is the only one. It costs as much as a primitive to build, but composes as freely as a macro. - -A request that is none of the three is refused, and the +- **A declaration section** is a block of declarations of one kind, such as + `variables:` or `given:`. One enters where it states something no section + states, where a file decides it without data, and where the typesetter prints + it. `given:` entered on all three. No other section says that a column + belongs to another file, and that is what lets a component file load and + print on its own. + +A request that is none of the four is refused, and the [table of refusals](#deliberate-non-primitives) records it with what to write instead. @@ -131,16 +137,22 @@ That another tool has a feature is not by itself a reason to add it. ## Composition (component libraries) -A component library is a set of templates, such as a boiler, a battery and a -line, that agree on how ports and flows are named. You merge the templates you -need into one file, wire the components together with a connectivity table in -the data, and close the system with one `sum(by=)` balance. +A component library is a set of fragments, one file per component type, such +as a boiler, a battery and a line. The fragments agree on how ports and flows +are named. You merge the fragments you need into one file, wire the components together +with a connectivity table in the data, and close the system with one `sum(by=)` +balance. The topology is data. Adding a second battery is a row in a table, so the file grows with the number of component _types_. +A fragment reads the coupling surface it is written against, and declares that +column under +[`given: variables:`](../reference/language/declarations.md#given). So it +loads on its own, and prints as math on its own. + Merging happens before `to_spec`. Every function here takes a `dict` as well as a path, so a model assembled in Python is checked exactly as a file is, and `Spec.to_yaml()` writes the file a reviewer reads. A `dict` may hold only what a -file may hold. A built-in merge, and namespaces so that two templates can each +file may hold. A built-in merge, and namespaces so that two fragments can each declare a `p`, are both things a library does before it hands over a `dict`. diff --git a/docs/reference/language/declarations.md b/docs/reference/language/declarations.md index f3bb9f71..33f0e67c 100644 --- a/docs/reference/language/declarations.md +++ b/docs/reference/language/declarations.md @@ -5,8 +5,9 @@ SPDX-License-Identifier: CC-BY-4.0 # Parameters, variables, constraints and the objective -These four blocks carry the math. Each takes an optional `description:`, free -text that the [typeset](../typeset.md#descriptions) legend prints. +These four blocks carry the math, and `given:` names what the math reads from +another file. Each takes an optional `description:`, free text that the +[typeset](../typeset.md#descriptions) legend prints. ## `parameters` @@ -84,6 +85,82 @@ a bound parameter are a subset of the variable's. Equal bounds pin a variable ([fix a quantity](../../howto/pin-a-variable.md)). A pinned variable is still a variable. +## `given` + +`given:` holds what this file reads and does not build: columns under +`variables:`, row families under `constraints:`. It takes those two keys and no +other. A file with a `given:` block loads and prints on its own. + +### `given: variables` + +A given variable is a column this file reads and another file introduces. + +```yaml +dimensions: + snapshot: { dtype: int } + port: { dtype: str } + generator: { dtype: str } +relations: + gen_port: { key: generator, values: port } +given: + variables: + flow: + dims: [snapshot, port] + description: what a port puts into its bus +variables: + gen_p: { dims: [snapshot, generator], bounds: { lower: 0 } } +constraints: + gen_injects: + dims: [snapshot, generator] + expression: at(flow, by=gen_port, over=port, into=generator) == gen_p +``` + +| Field | | | +| ------------- | ------------------------------------------------- | -------------------- | +| `dims` | required. The dimensions the column is indexed by | | +| `domain` | `continuous`, `integer` or `binary` | default `continuous` | +| `description` | free text | default `null` | + +There is no `bounds` and no `where`. The file that introduces the column owns +both. + +An expression reads a given variable as it reads any other. A name declared +under both `variables:` and `given: variables:` is refused. The typeset legend +lists a given variable under _Given_, and prints no domain line for it. + +Where nothing in this language introduces the column, the program carries the +declaration for a consumer to bind +([what a program does not build](../reading.md#what-a-program-does-not-build)). + +### `given: constraints` + +A given constraint is a row family this file reads the dual of and another +model builds. + +```yaml +dimensions: + snapshot: { dtype: int } + bus: { dtype: str } +given: + constraints: + balance: + dims: [snapshot, bus] + description: the host model clears each bus +expressions: + price: + expression: dual(balance) +``` + +| Field | | | +| ------------- | ------------------------------------------------- | -------------- | +| `dims` | required. The dimensions the row family runs over | | +| `description` | free text | default `null` | + +There is no `expression` and no `sense`, because nothing here builds the row. +`dual(name)` is the only place a given row family may be named, and the frame +gives the reported expression its dimensions. A name declared under both +`constraints:` and `given: constraints:` is refused. + ## `constraints` One block is one rule. The name of the block is the name of the constraint. diff --git a/docs/reference/language/errors.md b/docs/reference/language/errors.md index 0ea141d0..fe068674 100644 --- a/docs/reference/language/errors.md +++ b/docs/reference/language/errors.md @@ -34,6 +34,7 @@ loads. | `kind` | The file has… | The advice says… | | --------------- | --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `never-an-axis` | a dimension nothing is indexed by, nothing aggregates into and no relation targets | remove it, or keep it knowingly if its declarations are still to come | +| `given` | a column or a row family it reads and does not build ([given](declarations.md#given)) | a consumer binds it to the model this one is layered onto | | `unbounded` | a variable that no constraint uses, whose objective term pushes it towards a bound it does not have | give it a finite bound, or the constraint that was meant to define it | ```text diff --git a/docs/reference/language/file.md b/docs/reference/language/file.md index f27062a3..ec78198f 100644 --- a/docs/reference/language/file.md +++ b/docs/reference/language/file.md @@ -5,8 +5,8 @@ SPDX-License-Identifier: CC-BY-4.0 # File shape -A model file is a YAML mapping with **ten declaration keys**, plus `version` -and `description`. Any subset of the ten is accepted. +A model file is a YAML mapping with **eleven declaration keys**, plus `version` +and `description`. Any subset of the eleven is accepted. | Key | | | ------------- | ------------------------------------------------------------------------------------------------- | @@ -14,6 +14,7 @@ and `description`. Any subset of the ten is accepted. | `relations` | named relations between dimensions ([relations](relations.md)) | | `parameters` | the data the model expects ([declarations](declarations.md)) | | `variables` | what the solver decides | +| `given` | what this file reads and another file builds ([given](declarations.md#given)) | | `constraints` | the rules those decisions obey | | `objective` | what is minimised or maximised | | `expressions` | named quantities, reusable in the math and readable after a solve ([named expressions](named.md)) | diff --git a/docs/reference/language/index.md b/docs/reference/language/index.md index 07454afa..84eca1f3 100644 --- a/docs/reference/language/index.md +++ b/docs/reference/language/index.md @@ -42,7 +42,7 @@ That file is a complete model. The pages below give the exact rules. | | | | ----------------------------------------------------------------------- | ------------------------------------------------------------------------- | -| [File shape](file.md) | the ten keys, `version` and `description` | +| [File shape](file.md) | the eleven keys, `version` and `description` | | [Dimensions](dimensions.md) | the axes | | [Relations](relations.md) | the maps from one axis onto another | | [Parameters, variables, constraints and the objective](declarations.md) | the four blocks that carry the math | @@ -60,7 +60,7 @@ message that names the fix. These are the rules it checks. | # | Rule | | | --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | -| 1 | A file has ten declaration keys, plus `version` and `description`. An unknown key is refused, with the nearest valid key named. | [File shape](file.md) | +| 1 | A file has eleven declaration keys, plus `version` and `description`. An unknown key is refused, with the nearest valid key named. | [File shape](file.md) | | 2 | Everything that can be checked without data is checked when the file loads. | [Errors](errors.md) | | 3 | Every name is declared once. A parameter and a dimension both called `snapshot` is refused. | [Names](expressions.md#name-resolution) | | 4 | Where a name may stand depends on what it is. A dimension follows `over=` or `along=`, and is never multiplied. | [Names](expressions.md#name-resolution) | diff --git a/docs/reference/reading.md b/docs/reference/reading.md index 7d26b4b4..750c675b 100644 --- a/docs/reference/reading.md +++ b/docs/reference/reading.md @@ -100,6 +100,38 @@ A predicate you build yourself answers the same four questions: wrap it in so a boolean literal stands at a mask's root or nowhere. A `Region`'s `when` arrives as a `Mask` too. The node classes live in `math_spec.program`. +## What a program does not build + +`program.given.variables` and `program.given.constraints` name what the model +reads and does not build ([given](language/declarations.md#given)). Every +other group is a build instruction. These two are names to look up in the +model this one is layered onto. + +```python +layer = to_program( + { + 'dimensions': {'snapshot': {'dtype': 'int'}, 'bus': {'dtype': 'str'}}, + 'given': { + 'variables': {'p': {'dims': ['snapshot', 'bus']}}, + 'constraints': {'balance': {'dims': ['snapshot', 'bus']}}, + }, + 'parameters': {'rate': {'dims': ['bus']}}, + 'constraints': {'cap': {'dims': [], 'expression': 'sum(p * rate) <= 100'}}, + 'expressions': {'price': {'expression': 'dual(balance)'}}, + } +) + +sorted(layer.variables) # [] +sorted(layer.given.variables) # ['p'] +layer.given.constraints['balance'].dims # ('snapshot', 'bus') +``` + +A consumer that builds the program binds each name to a column or a row family +the host model holds. It checks that the frame matches, and refuses what it +cannot bind. A consumer with no host refuses a program whose two groups are not +both empty. `advice` returns one note of kind `given` per name +([what `advice` warns about](language/errors.md#what-advice-warns-about)). + ## Asking what a program uses `program.footprint` says which of the language's constructs one model uses. diff --git a/schema/math-spec.schema.json b/schema/math-spec.schema.json index 2cdc05fa..1aee172f 100644 --- a/schema/math-spec.schema.json +++ b/schema/math-spec.schema.json @@ -216,6 +216,100 @@ "title": "ExpressionCase", "type": "object" }, + "GivenBlock": { + "additionalProperties": false, + "description": "What this file reads and does not build, by kind. Closed at the two kinds.", + "properties": { + "constraints": { + "additionalProperties": { + "$ref": "#/$defs/GivenConstraintBlock" + }, + "default": {}, + "title": "Constraints", + "type": "object" + }, + "variables": { + "additionalProperties": { + "$ref": "#/$defs/GivenVariableBlock" + }, + "default": {}, + "title": "Variables", + "type": "object" + } + }, + "title": "GivenBlock", + "type": "object" + }, + "GivenConstraintBlock": { + "additionalProperties": false, + "description": "A row family this file reads the dual of and another model builds.\n\nThe frame says how many duals there are and what indexes them, which is\nwhat ``dual()`` needs. There is no ``expression:``, because nothing here\nbuilds a row.", + "properties": { + "description": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Description" + }, + "dims": { + "items": { + "type": "string" + }, + "title": "Dims", + "type": "array" + } + }, + "required": [ + "dims" + ], + "title": "GivenConstraintBlock", + "type": "object" + }, + "GivenVariableBlock": { + "additionalProperties": false, + "description": "A column this file reads and another file introduces.\n\nThe frame and the domain are all this file states. The file that introduces\nthe column owns its bounds and its mask.", + "properties": { + "description": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Description" + }, + "dims": { + "items": { + "type": "string" + }, + "title": "Dims", + "type": "array" + }, + "domain": { + "default": "continuous", + "enum": [ + "continuous", + "integer", + "binary" + ], + "title": "Domain", + "type": "string" + } + }, + "required": [ + "dims" + ], + "title": "GivenVariableBlock", + "type": "object" + }, "MacroBlock": { "additionalProperties": false, "description": "A parameterised expression template, defined in the YAML itself.\n\nLanguage, not code: formals (``args`` positional, ``kwargs`` keyword)\nshadow model names inside the template, and every call site expands into\ncore AST before either backend sees the expression.", @@ -636,7 +730,7 @@ }, "$schema": "https://json-schema.org/draft/2020-12/schema", "additionalProperties": false, - "description": "The declared math \u2014 one YAML file, or one dict, validated. Nothing here has seen data.\n\nA ``Spec`` that exists has passed the whole language: constructing one by\nany route \u2014 ``to_spec``, :meth:`model_validate`, the constructor \u2014 runs\nevery load-time check, expansion and expression pass included, and raises\n:class:`~math_spec.errors.LanguageError` on a model the language refuses.\nHolding one is the proof, so nothing downstream checks it again.\n\nThe API is the ten declaration sections plus ``version`` and\n``description``, and two ways back out: :meth:`to_dict` for the model as\ndata, :meth:`to_yaml` for the file a reviewer reads. Everything else on\nthis class is pydantic's, not a contract this package keeps.", + "description": "The declared math \u2014 one YAML file, or one dict, validated. Nothing here has seen data.\n\nA ``Spec`` that exists has passed the whole language: constructing one by\nany route \u2014 ``to_spec``, :meth:`model_validate`, the constructor \u2014 runs\nevery load-time check, expansion and expression pass included, and raises\n:class:`~math_spec.errors.LanguageError` on a model the language refuses.\nHolding one is the proof, so nothing downstream checks it again.\n\nThe API is the eleven declaration sections plus ``version`` and\n``description``, and two ways back out: :meth:`to_dict` for the model as\ndata, :meth:`to_yaml` for the file a reviewer reads. Everything else on\nthis class is pydantic's, not a contract this package keeps.", "properties": { "constraints": { "additionalProperties": { @@ -674,6 +768,13 @@ "title": "Expressions", "type": "object" }, + "given": { + "$ref": "#/$defs/GivenBlock", + "default": { + "constraints": {}, + "variables": {} + } + }, "macros": { "additionalProperties": { "$ref": "#/$defs/MacroBlock" diff --git a/src/math_spec/advice.py b/src/math_spec/advice.py index 910478db..1bc0035b 100644 --- a/src/math_spec/advice.py +++ b/src/math_spec/advice.py @@ -33,11 +33,30 @@ def advice(model: str | Path | dict[str, Any] | Spec | Program) -> tuple[Advice, answer alike. Returns: - The never-an-axis advice in declaration order, then the unboundedness - advice; ``str()`` of each is its sentence. + The never-an-axis advice in declaration order, then one note per + declaration the program reads and does not build, then the + unboundedness advice; ``str()`` of each is its sentence. """ program = to_program(model) - return tuple(_never_an_axis(program) + unbounded_notes(program)) + return tuple(_never_an_axis(program) + _given(program) + unbounded_notes(program)) + + +def _given(program: Program) -> list[Advice]: + """One note per declaration the program reads and does not build. + + A note rather than a refusal: the file is a model somebody meant, and only + the consumer can tell whether it holds a host to bind the name to. + """ + return [ + Advice( + 'given', + name, + f"{kind} '{name}' is read here and built elsewhere: a consumer binds it to the model this " + f'one is layered onto, checks the frame, and refuses where it cannot bind it.', + ) + for kind, group in (('variable', program.given.variables), ('row family', program.given.constraints)) + for name in group + ] def _never_an_axis(program: Program) -> list[Advice]: @@ -45,10 +64,18 @@ def _never_an_axis(program: Program) -> list[Advice]: A dimension a relation has a column over is reached: its members are the labels that column is checked against, and a ``where`` selects on them, - so it is in use even where nothing is indexed by it. + so it is in use even where nothing is indexed by it. A dimension only a + given declaration indexes is reached too: the column exists, in another + file. """ reached: set[str] = set() - for declaration in (*program.parameters.values(), *program.variables.values(), *program.constraints.values()): + for declaration in ( + *program.parameters.values(), + *program.variables.values(), + *program.constraints.values(), + *program.given.variables.values(), + *program.given.constraints.values(), + ): reached.update(declaration.dims) reached |= _produced_axes(program) reached |= {dim for lk in program.relations.values() for dim in lk.dims} diff --git a/src/math_spec/boundedness.py b/src/math_spec/boundedness.py index 0c6b3e7e..7c2aa48c 100644 --- a/src/math_spec/boundedness.py +++ b/src/math_spec/boundedness.py @@ -78,7 +78,7 @@ def unbounded_notes(program: Program) -> list[Advice]: minimize = program.objective.sense == 'minimize' notes: list[Advice] = [] for vname, sign in signs.items(): - if sign is None or vname in constrained: + if sign is None or vname in constrained or vname in program.given.variables: continue side: BoundSide = 'lower' if minimize == (sign == '+') else 'upper' if _is_open(program.variables[vname], side): diff --git a/src/math_spec/dimensions.py b/src/math_spec/dimensions.py index 29d1b514..222a3a5d 100644 --- a/src/math_spec/dimensions.py +++ b/src/math_spec/dimensions.py @@ -90,14 +90,14 @@ def _dims( return frozenset(schema.parameters[node.name].dims) if isinstance(node, VariableNode): - return frozenset(schema.variables[node.name].dims) + return frozenset({**schema.variables, **schema.given.variables}[node.name].dims) if isinstance(node, UnresolvedNode | KwargNode): msg = f'{type(node).__name__} reached the dim checker; resolve the expression first.' raise AssertionError(msg) if isinstance(node, DualNode): - return frozenset(schema.constraints[node.constraint].dims) + return frozenset({**schema.constraints, **schema.given.constraints}[node.constraint].dims) if isinstance(node, FunctionCallNode): return _dims_call(node, schema, context) diff --git a/src/math_spec/errors.py b/src/math_spec/errors.py index 1c467ae9..3daa82b0 100644 --- a/src/math_spec/errors.py +++ b/src/math_spec/errors.py @@ -18,7 +18,7 @@ #: Which pass an :class:`Advice` comes from. Closed, like the operator set: a #: consumer filtering on it can enumerate every value. -AdviceKind = Literal['never-an-axis', 'unbounded'] +AdviceKind = Literal['never-an-axis', 'given', 'unbounded'] ADVICE_KINDS = frozenset(get_args(AdviceKind)) diff --git a/src/math_spec/lowering.py b/src/math_spec/lowering.py index 6cca5857..dc3e3a59 100644 --- a/src/math_spec/lowering.py +++ b/src/math_spec/lowering.py @@ -165,6 +165,10 @@ def lower_program(expanded: _ExpandedSpec) -> program.Program: expressions[name] = program.ExpressionDeclaration( _Lowering(expanded, f"named expression '{name}'").expr(ast), in_math=name in resolved.read_by_the_math ) + given = program.GivenTargets( + variables={name: program.GivenDeclaration(tuple(g.dims)) for name, g in expanded.given.variables.items()}, + constraints={name: program.GivenDeclaration(tuple(g.dims)) for name, g in expanded.given.constraints.items()}, + ) return program.Program( parameters=parameters, variables=variables, @@ -174,6 +178,7 @@ def lower_program(expanded: _ExpandedSpec) -> program.Program: sos=sos, piecewise={name: declaration_of(ex) for name, ex in expanded.expanded_piecewise.items()}, named_expressions=expressions, + given=given, ) diff --git a/src/math_spec/model.py b/src/math_spec/model.py index 54a8845d..15979c7a 100644 --- a/src/math_spec/model.py +++ b/src/math_spec/model.py @@ -308,6 +308,49 @@ def _absence_needs_a_mask(self) -> VariableBlock: return self +class GivenVariableBlock(_StrictBlock): + """A column this file reads and another file introduces. + + The frame and the domain are all this file states. The file that introduces + the column owns its bounds and its mask. + """ + + _label: ClassVar[str] = 'a given variable declaration' + + dims: list[str] + domain: VariableDomain = 'continuous' + description: str | None = None + + +class GivenConstraintBlock(_StrictBlock): + """A row family this file reads the dual of and another model builds. + + The frame says how many duals there are and what indexes them, which is + what ``dual()`` needs. There is no ``expression:``, because nothing here + builds a row. + """ + + _label: ClassVar[str] = 'a given constraint declaration' + + dims: list[str] + description: str | None = None + + +class GivenBlock(_StrictBlock): + """What this file reads and does not build, by kind. Closed at the two kinds.""" + + _label: ClassVar[str] = 'a given block' + + #: Columns another file introduces (:class:`GivenVariableBlock`). + variables: dict[str, GivenVariableBlock] = {} + #: Row families another model builds (:class:`GivenConstraintBlock`). + constraints: dict[str, GivenConstraintBlock] = {} + + def __bool__(self) -> bool: + """Whether the file reads anything it does not build.""" + return bool(self.variables or self.constraints) + + class ConstraintBlock(_StrictBlock): """A declared constraint: one rule, over one frame.""" @@ -693,7 +736,7 @@ class Spec(_StrictBlock): :class:`~math_spec.errors.LanguageError` on a model the language refuses. Holding one is the proof, so nothing downstream checks it again. - The API is the ten declaration sections plus ``version`` and + The API is the eleven declaration sections plus ``version`` and ``description``, and two ways back out: :meth:`to_dict` for the model as data, :meth:`to_yaml` for the file a reviewer reads. Everything else on this class is pydantic's, not a contract this package keeps. @@ -724,6 +767,10 @@ class Spec(_StrictBlock): macros: dict[str, MacroBlock] = {} piecewise: dict[str, PiecewiseBlock] = {} sos: dict[str, SosBlock] = {} + #: What this file reads and does not build (:class:`GivenBlock`): columns + #: under ``variables:``, row families under ``constraints:``. Empty in a + #: file that stands alone. + given: GivenBlock = GivenBlock() def relations_of(self, dimension: str) -> dict[str, RelationBlock]: """The relations with a column over *dimension*, by name.""" @@ -783,13 +830,16 @@ def _names_are_names(self) -> Spec: Read off the model's own mappings rather than a list of sections, so a section added later cannot be forgotten here — every mapping a Spec - carries is keyed by a declaration name. + carries is keyed by a declaration name. ``given:`` nests its two + mappings one level down, so they are read off :class:`GivenBlock` the + same way. """ + sections = [*self, *((f'given: {kind}', group) for kind, group in self.given)] errors = [ f'{section}: {name!r} is not a name. A declaration is named the way an expression ' f'writes it — a letter or an underscore, then letters, digits or underscores — so ' f'nothing can refer to this one. Rename it.' - for section, value in self + for section, value in sections if isinstance(value, dict) for name in value if not re.fullmatch(NAME, name) @@ -808,11 +858,25 @@ def _validate_references(self) -> Spec: *self._relation_targets(), *self._bound_names(), *self._sos_shapes(), + *self._given_constraint_collisions(), ] if errors: raise ValueError('\n'.join(errors)) return self + def _given_constraint_collisions(self) -> Iterator[str]: + """A row family is either built here or given, never both. + + Constraint names sit outside the flat namespace :meth:`_name_collisions` + walks, so this is the one place the two constraint sections meet. + """ + for name in self.given.constraints: + if name in self.constraints: + yield ( + f"Given constraint '{name}' is also declared under 'constraints:'. A row family is " + f'either built by this file or given to it — drop one of the two.' + ) + def _name_collisions(self) -> Iterator[str]: """A name is declared once, and never as a built-in operator.""" kinds: list[tuple[str, Iterable[str]]] = [ @@ -820,6 +884,7 @@ def _name_collisions(self) -> Iterator[str]: ('relation', self.relations), ('parameter', self.parameters), ('variable', self.variables), + ('given variable', self.given.variables), ('named expression', self.expressions), ('macro', self.macros), ] @@ -845,6 +910,8 @@ def _frame_dimensions(self) -> Iterator[str]: frames = [ *(('Parameter', name, p.dims) for name, p in self.parameters.items()), *(('Variable', name, v.dims) for name, v in self.variables.items()), + *(('Given variable', name, g.dims) for name, g in self.given.variables.items()), + *(('Given constraint', name, g.dims) for name, g in self.given.constraints.items()), *(('Constraint', name, c.dims) for name, c in self.constraints.items()), *(('Named expression', name, e.dims or []) for name, e in self.expressions.items()), ] diff --git a/src/math_spec/program.py b/src/math_spec/program.py index 42b59f51..7a35ebc2 100644 --- a/src/math_spec/program.py +++ b/src/math_spec/program.py @@ -64,6 +64,8 @@ 'FanIn', 'FirstOf', 'Footprint', + 'GivenDeclaration', + 'GivenTargets', 'GroupSum', 'Increasing', 'LastOf', @@ -746,6 +748,40 @@ class VariableDeclaration: absence: VariableAbsence = 'undefined' +@dataclass(frozen=True) +class GivenDeclaration: + """A column or a row family this program reads and does not build. + + The frame is the whole declaration. A consumer looks the name up in the + model this one is layered onto, checks the frame against what it finds, + and refuses what it cannot bind. + """ + + dims: tuple[str, ...] + + +@dataclass(frozen=True) +class GivenTargets: + """What a program reads and does not build, by kind. + + Both groups are empty in a program built from one whole model. Both are + sealed at construction, like every group of :class:`Program`. + """ + + #: Columns to bind, by name. + variables: Mapping[str, GivenDeclaration] = Sealed({}) + #: Row families to bind, by name, read back after the solve. + constraints: Mapping[str, GivenDeclaration] = Sealed({}) + + def __post_init__(self) -> None: + for f in fields(self): + object.__setattr__(self, f.name, Sealed(getattr(self, f.name))) + + def __bool__(self) -> bool: + """Whether the program reads anything it does not build.""" + return bool(self.variables or self.constraints) + + @dataclass(frozen=True) class ConstraintDeclaration: """``lhs sense rhs`` for each coord combination of ``dims``. @@ -978,6 +1014,10 @@ class Program: #: named expression is outside the language is refused by every verb that #: reads the file rather than only by the one that reads the expression. named_expressions: Mapping[str, ExpressionDeclaration] = Sealed({}) + #: What this program reads and does not build (:class:`GivenTargets`). A + #: consumer binds each name to what the model it is layered onto holds; + #: nothing here emits a column or a row. + given: GivenTargets = GivenTargets() def __post_init__(self) -> None: """Seal every group, so a program handed out cannot be written to.""" diff --git a/src/math_spec/resolution.py b/src/math_spec/resolution.py index 48325bcc..5708493b 100644 --- a/src/math_spec/resolution.py +++ b/src/math_spec/resolution.py @@ -132,8 +132,9 @@ def __init__( @classmethod def of(cls, schema: Spec) -> Namespace: """Build the namespace of *schema*, the whole of what a file may name.""" + variables = {**schema.variables, **schema.given.variables} return cls( - schema.variables, + variables, schema.parameters, schema.dimensions, {n: RelationDeclaration(n, lk.pairs, lk.key_roles) for n, lk in schema.relations.items()}, @@ -143,9 +144,9 @@ def of(cls, schema: Spec) -> Namespace: }, { **{p: tuple(pd.dims) for p, pd in schema.parameters.items()}, - **{v: tuple(vd.dims) for v, vd in schema.variables.items()}, + **{v: tuple(vd.dims) for v, vd in variables.items()}, }, - schema.constraints, + {**schema.constraints, **schema.given.constraints}, ) def kind(self, name: str) -> DeclarationKind | None: @@ -191,7 +192,8 @@ def unknown_constraint(self, name: str, context: str, *, formals: Iterable[str] return ( f"{context}: dual({name}): '{name}' is not a declared constraint{also}.\n" f' Constraints: {sorted(self.constraints)}\n' - f"Check for typos, or declare '{name}' under 'constraints:'." + f"Check for typos, or declare '{name}': under 'constraints:' if this file builds the row, " + f"or under 'given: constraints:' if it reads the dual of a row another model builds." ) diff --git a/src/math_spec/typesetting/symbols.py b/src/math_spec/typesetting/symbols.py index c6fb99d9..ae0bf154 100644 --- a/src/math_spec/typesetting/symbols.py +++ b/src/math_spec/typesetting/symbols.py @@ -108,8 +108,8 @@ def __init__(self, schema: _ExpandedSpec, fmt: Format, table: SymbolTable) -> No f'and nothing translates between notations — write a {fmt.notation} table.' ) raise SchemaError(msg) - chosen = frozenset(schema.variables) | chosen_expressions(schema) - names = (*schema.parameters, *schema.variables, *schema.expressions) + chosen = frozenset(schema.variables) | frozenset(schema.given.variables) | chosen_expressions(schema) + names = (*schema.parameters, *schema.variables, *schema.given.variables, *schema.expressions) declared = frozenset(names) #: Names the table spelled; the convention note quotes only derived symbols. @@ -129,7 +129,7 @@ def __init__(self, schema: _ExpandedSpec, fmt: Format, table: SymbolTable) -> No #: overrides it. self.constraint: dict[str, str] = { name: table.names[name] if name in table.names else _derive_name_symbol(name, declared, fmt, given=True) - for name in schema.constraints + for name in (*schema.constraints, *schema.given.constraints) } self.index: dict[str, str] = {} @@ -239,7 +239,13 @@ def checked_against(self, schema: _ExpandedSpec) -> SymbolTable: """Reject entries naming nothing in *schema*, with the near miss.""" dims = set(schema.dimensions) everything = ( - dims | set(schema.parameters) | set(schema.variables) | set(schema.expressions) | set(schema.constraints) + dims + | set(schema.parameters) + | set(schema.variables) + | set(schema.given.variables) + | set(schema.expressions) + | set(schema.constraints) + | set(schema.given.constraints) ) errors = [ *(_unknown_entry(d, 'dimensions', dims) for d in {*self.indices, *self.sets} - dims), diff --git a/src/math_spec/typesetting/walk.py b/src/math_spec/typesetting/walk.py index bedc65e1..acc97626 100644 --- a/src/math_spec/typesetting/walk.py +++ b/src/math_spec/typesetting/walk.py @@ -356,7 +356,8 @@ def _arithmetic(self, node: ArithmeticNode, ctx: _Context) -> tuple[str, int]: return ctx.indexed(self.symbols.name[node.name], list(self.schema.parameters[node.name].dims)), _ATOM if isinstance(node, VariableNode): - return ctx.indexed(self.symbols.name[node.name], list(self.schema.variables[node.name].dims)), _ATOM + frames = {**self.schema.variables, **self.schema.given.variables} + return ctx.indexed(self.symbols.name[node.name], list(frames[node.name].dims)), _ATOM if isinstance(node, UnaryOperatorNode): if node.op == '+': @@ -867,6 +868,20 @@ def glossaries(self, noticed: Noticed) -> list[Glossary]: self._entry(self.symbols.name[v], f'{fmt.mono(v)}{self._over(list(block.dims))}', block.description) for v, block in self.schema.variables.items() ] + given = [ + *( + self._entry(self.symbols.name[g], f'{fmt.mono(g)}{self._over(list(block.dims))}', block.description) + for g, block in self.schema.given.variables.items() + ), + *( + self._entry( + self.symbols.constraint[g], + f'{fmt.mono(g)}{self._over(list(block.dims))}, a row family this file reads the dual of', + block.description, + ) + for g, block in self.schema.given.constraints.items() + ), + ] definitions = [ self._entry(self.symbols.name[e], f'{fmt.mono(e)}{self._over(self.frames[e])}', block.description) for e, block in self.schema.expressions.items() @@ -876,6 +891,7 @@ def glossaries(self, noticed: Noticed) -> list[Glossary]: Glossary('Sets', sets), Glossary('Parameters', parameters), Glossary('Variables', variables), + Glossary('Given', given), Glossary('Definitions', definitions), ) return [group for group in groups if group.entries] diff --git a/tests/test_advice.py b/tests/test_advice.py index 37232fd1..17f46f25 100644 --- a/tests/test_advice.py +++ b/tests/test_advice.py @@ -66,7 +66,28 @@ def test_both_kinds_of_note_come_through_the_one_door(): assert [(n.kind, n.subject) for n in notes] == [('never-an-axis', 'h'), ('unbounded', 'p')], ( 'the never-an-axis advice comes first, then the unboundedness advice' ) - assert {n.kind for n in notes} == ADVICE_KINDS, 'every kind a consumer can pin against is one this file produces' + + +#: A model whose only note is the third kind: `flow` is a column this file +#: reads and whatever it is layered onto builds. `p` is bounded on both sides +#: and every dimension is indexed, so neither other pass has anything to say. +READS_A_COLUMN = { + 'dimensions': {'g': {'dtype': 'str'}}, + 'given': {'variables': {'flow': {'dims': ['g']}}}, + 'variables': {'p': {'dims': ['g'], 'bounds': {'lower': 0, 'upper': 1}}}, + 'constraints': {'tie': {'dims': ['g'], 'expression': 'p == flow'}}, +} + + +def test_a_column_read_and_not_built_is_advised(): + (note,) = advice(READS_A_COLUMN) + assert (note.kind, note.subject) == ('given', 'flow') + assert 'binds it to the model' in str(note), 'the note says whose job the column is' + + +def test_every_kind_a_consumer_can_pin_against_is_produced_here(): + kinds = {note.kind for note in (*advice(BOTH_KINDS), *advice(READS_A_COLUMN))} + assert kinds == ADVICE_KINDS, 'every kind a consumer can pin against is one these fixtures produce' def _written(model: dict, tmp_path: Path) -> Path: diff --git a/tests/test_given.py b/tests/test_given.py new file mode 100644 index 00000000..02ccf066 --- /dev/null +++ b/tests/test_given.py @@ -0,0 +1,217 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +"""What a file reads and does not build: a column, and a row family. + +A fragment reads a column the file beside it introduces. A layer reads a +column, or the dual of a row family, that a model outside the language holds. +What both need is that the file stands on its own: it loads, it lowers, and it +prints as math, without the thing that owns what it reads. +""" + +from __future__ import annotations + +import pytest + +from math_spec import FORMATS, LanguageError, advice, to_markdown, to_program, to_spec, typeset + +#: One component file: it pins the flow at its own port, and the column it +#: pins belongs to another fragment. +SUPPLY = { + 'description': 'A fleet of generators, each on one port.', + 'dimensions': {'snapshot': {'dtype': 'int'}, 'port': {'dtype': 'str'}, 'generator': {'dtype': 'str'}}, + 'relations': {'gen_port': {'key': 'generator', 'values': 'port'}}, + 'given': {'variables': {'flow': {'dims': ['snapshot', 'port'], 'description': 'what a port puts into its bus'}}}, + 'parameters': {'gen_cost': {'dims': ['generator']}, 'gen_p_max': {'dims': ['generator']}}, + 'variables': {'gen_p': {'dims': ['snapshot', 'generator'], 'bounds': {'lower': 0, 'upper': 'gen_p_max'}}}, + 'constraints': { + 'gen_injects': { + 'dims': ['snapshot', 'generator'], + 'expression': 'at(flow, by=gen_port, over=port, into=generator) == gen_p', + } + }, + 'objective': {'sense': 'minimize', 'expression': 'sum(gen_p * gen_cost)'}, +} + + +def test_given_holds_two_kinds_and_refuses_a_third(): + """The section is closed, so a kind nobody has admitted yet is the schema's own refusal.""" + with pytest.raises(LanguageError) as raised: + to_spec({**SUPPLY, 'given': {'parameters': {'gen_cost': {'dims': ['generator']}}}}) + assert 'Valid keys: constraints, variables' in str(raised.value), 'the refusal names what the block takes' + + +def test_a_fragment_that_says_what_it_reads_loads_on_its_own(): + spec = to_spec(SUPPLY) + assert sorted(spec.given.variables) == ['flow'], 'the column it reads is a declaration like any other' + assert sorted(spec.variables) == ['gen_p'], 'and it is not one of the columns this file introduces' + + +def test_a_given_name_is_held_to_the_name_rule(): + """`_names_are_names` walks the top-level mappings, and `given:` nests its two one level down.""" + with pytest.raises(LanguageError, match=r"given: variables: 'no-flow' is not a name"): + to_spec({**SUPPLY, 'given': {'variables': {'no-flow': {'dims': ['snapshot', 'port']}}}}) + + +def test_a_whole_model_writes_no_given_block(): + whole = to_spec({**SUPPLY, 'given': {}, 'constraints': {}}) + assert 'given' not in whole.to_dict(), 'an empty section is an absence, and is left out' + + +def test_a_fragment_round_trips_through_its_own_data(): + spec = to_spec(SUPPLY) + assert to_spec(spec.to_dict()) == spec + + +@pytest.mark.parametrize('fmt', sorted(FORMATS)) +def test_a_fragment_prints_as_math_in_every_format(fmt): + assert typeset(SUPPLY, fmt), f'{fmt} rendered nothing' + + +def test_the_given_column_prints_under_its_own_heading(): + printed = to_markdown(SUPPLY) + assert '#### Given' in printed, 'the legend says which symbols the file does not introduce' + assert '`flow`' in printed.split('#### Given')[1] + + +def test_a_program_carries_the_column_it_reads_apart_from_the_ones_it_builds(): + """The distinction a builder needs: create this column, or bind it to one the host already holds.""" + program = to_program(SUPPLY) + assert sorted(program.variables) == ['gen_p'], 'a build reads this group and creates a column for each' + assert sorted(program.given.variables) == ['flow'], 'and binds each of these to a column it is given' + assert program.given.variables['flow'].dims == ('snapshot', 'port'), 'the frame is what a binder checks' + + +def test_what_a_program_reads_is_sealed_like_what_it_builds(): + program = to_program(SUPPLY) + with pytest.raises(TypeError, match='does not support item assignment'): + program.given.variables['p'] = program.given.variables['flow'] + + +def test_the_advice_says_which_columns_a_consumer_has_to_bind(): + (note,) = [note for note in advice(SUPPLY) if note.kind == 'given'] + assert note.subject == 'flow' + + +def test_a_given_column_in_the_objective_is_not_advised_unbounded(): + """The unboundedness pass reads a variable's bounds, and a given column's bounds are the owner's.""" + priced = {**SUPPLY, 'constraints': {}, 'objective': {'sense': 'minimize', 'expression': 'sum(flow)'}} + assert not [note for note in advice(priced) if note.kind == 'unbounded'] + + +def test_a_name_both_introduced_and_given_in_one_file_is_refused(): + both = {**SUPPLY, 'variables': {**SUPPLY['variables'], 'flow': {'dims': ['snapshot', 'port']}}} + with pytest.raises(LanguageError, match=r"Given variable 'flow' collides with the variable"): + to_spec(both) + + +@pytest.mark.parametrize( + ('block', 'says'), + [ + pytest.param({'dims': ['snapshot', 'nowhere']}, 'nowhere', id='a-frame-over-an-undeclared-dimension'), + pytest.param({'dims': ['snapshot', 'snapshot']}, 'twice', id='a-frame-naming-one-dimension-twice'), + pytest.param({'dims': ['snapshot'], 'bounds': {'lower': 0}}, 'bounds', id='bounds-the-owner-holds'), + pytest.param({'dims': ['snapshot'], 'where': 'gen_cost > 0'}, 'where', id='a-mask-the-owner-holds'), + ], +) +def test_a_given_declaration_is_refused_where_it_oversteps(block, says): + with pytest.raises(LanguageError) as raised: + to_spec({**SUPPLY, 'given': {'variables': {'flow': block}}}) + assert says in str(raised.value) + + +def test_an_expression_reads_a_given_column_as_it_reads_any_other(): + """Resolution and the dim algebra see one namespace, so the walk lands on the generator frame.""" + spec = to_spec(SUPPLY) + assert spec.constraints['gen_injects'].dims == ['snapshot', 'generator'] + + +#: A layer over a model this language never sees: it reads a column and the +#: dual of a row family, and adds one constraint of its own. +LAYER = { + 'description': 'A carbon cap laid over a model that already exists.', + 'dimensions': {'snapshot': {'dtype': 'int'}, 'bus': {'dtype': 'str'}}, + 'given': { + 'variables': {'p': {'dims': ['snapshot', 'bus']}}, + 'constraints': {'balance': {'dims': ['snapshot', 'bus'], 'description': 'the host clears each bus'}}, + }, + 'parameters': {'rate': {'dims': ['bus']}}, + 'constraints': {'cap': {'dims': [], 'expression': 'sum(p * rate) <= 100'}}, + 'expressions': {'price': {'expression': 'dual(balance)'}}, +} + + +def test_a_dual_may_name_a_row_family_this_file_does_not_build(): + spec = to_spec(LAYER) + assert sorted(spec.given.constraints) == ['balance'] + assert sorted(spec.constraints) == ['cap'], 'the row families it builds are its own, and that is not one' + + +def test_the_program_carries_the_row_family_a_consumer_binds(): + program = to_program(LAYER) + assert sorted(program.given.constraints) == ['balance'] + assert program.given.constraints['balance'].dims == ('snapshot', 'bus'), 'the frame is what a binder checks' + + +def test_the_dual_takes_its_frame_from_the_given_declaration(): + """Without the frame the reported expression has no dims, and nothing downstream could shape it.""" + assert to_markdown(LAYER).count(r'\lambda_{\mathrm{balance},t,b}') == 1 + + +def test_a_given_row_family_prints_under_the_given_heading(): + given = to_markdown(LAYER).split('#### Given')[1] + assert '`balance`' in given + assert 'reads the dual of' in given, 'the legend says what the file may do with it' + + +def test_a_row_family_both_built_and_given_is_refused(): + both = {**LAYER, 'constraints': {**LAYER['constraints'], 'balance': {'dims': [], 'expression': 'sum(p) >= 0'}}} + with pytest.raises(LanguageError, match=r"'balance'.*either built by this file or given to it"): + to_spec(both) + + +def test_a_dual_naming_nothing_says_where_to_declare_it(): + mistyped = {**LAYER, 'expressions': {'price': {'expression': 'dual(balnce)'}}} + with pytest.raises(LanguageError) as raised: + to_spec(mistyped) + message = str(raised.value) + assert "'constraints:'" in message and "'given: constraints:'" in message, ( + 'the message names both places the row family could be declared' + ) + + +@pytest.mark.parametrize( + ('block', 'says'), + [ + pytest.param({'dims': [], 'expression': 'sum(p) >= 0'}, 'expression', id='a-body-the-owner-holds'), + pytest.param({'dims': [], 'sense': '<='}, 'sense', id='a-sense-nothing-here-could-check'), + ], +) +def test_a_given_row_family_is_refused_where_it_oversteps(block, says): + with pytest.raises(LanguageError) as raised: + to_spec({**LAYER, 'given': {**LAYER['given'], 'constraints': {'balance': block}}}) + assert says in str(raised.value) + + +def test_the_advice_names_every_declaration_a_consumer_has_to_bind(): + subjects = {note.subject for note in advice(LAYER) if note.kind == 'given'} + assert subjects == {'p', 'balance'}, 'both the column and the row family are named' + + +#: `port` is named by nothing but the given column's frame, and `bus` by +#: nothing but the given row family's, so each is in use only through a +#: declaration this file does not build. +REACHED_ONLY_BY_A_GIVEN_FRAME = { + 'dimensions': {'g': {'dtype': 'str'}, 'port': {'dtype': 'str'}, 'bus': {'dtype': 'str'}}, + 'given': {'variables': {'flow': {'dims': ['port']}}, 'constraints': {'balance': {'dims': ['bus']}}}, + 'variables': {'p': {'dims': ['g'], 'bounds': {'lower': 0, 'upper': 1}}}, + 'constraints': {'tie': {'dims': ['g'], 'expression': 'p >= sum(flow, over=port)'}}, + 'expressions': {'price': {'expression': 'dual(balance)'}}, +} + + +def test_a_dimension_only_a_given_declaration_indexes_is_in_use(): + """The never-an-axis pass reads the frames a build emits, and these two are in neither.""" + unreached = {note.subject for note in advice(REACHED_ONLY_BY_A_GIVEN_FRAME) if note.kind == 'never-an-axis'} + assert not unreached, 'a dimension a given column or row family is indexed by is used' diff --git a/tests/test_reading_page.py b/tests/test_reading_page.py index 60725ac6..045ed6a9 100644 --- a/tests/test_reading_page.py +++ b/tests/test_reading_page.py @@ -55,6 +55,6 @@ def test_the_page_shows_the_declarations_the_expansion_emits(tmp_path, monkeypat exec(compile(code, str(PAGE), 'exec'), namespace) claims.extend(_claims(code)) - assert len(claims) == 12, 'every `expression # value` line on the page is checked; one without one is not' + assert len(claims) == 15, 'every `expression # value` line on the page is checked; one without one is not' for expression, claimed in claims: assert eval(expression, namespace) == claimed, f'reading.md says `{expression}` is {claimed}' From 84c7797994ab4bdc7cb383785d5ebb44f0d16ef7 Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 16:59:31 +0200 Subject: [PATCH 2/9] fix(language): typeset_declaration says why a given name prints no line, and given: sits beside variables: A given name asked of typeset_declaration was refused with "is not a named expression, constraint or variable". It now says that a given declaration prints no line of its own, and that it prints in the legend under Given. The `given` field moves after `variables` on Spec, so to_yaml writes the key where the file-shape table and the examples already put it. The schema and the golden output are unchanged. The symbols kwarg `given` is renamed to `upright`: it means the glyph, not the language's `given:` key. `balnce`, the did-you-mean input in tests/test_given.py, joins the typos allowance. --- docs/reference/language/declarations.md | 6 ++--- pyproject.toml | 5 +++-- src/math_spec/model.py | 8 +++---- src/math_spec/typesetting/__init__.py | 14 ++++++++++-- src/math_spec/typesetting/symbols.py | 18 +++++++-------- tests/typesetting/test_declaration.py | 30 ++++++++++++++++++++----- tests/typesetting/test_walk.py | 2 +- 7 files changed, 56 insertions(+), 27 deletions(-) diff --git a/docs/reference/language/declarations.md b/docs/reference/language/declarations.md index 33f0e67c..d183c0aa 100644 --- a/docs/reference/language/declarations.md +++ b/docs/reference/language/declarations.md @@ -102,13 +102,13 @@ dimensions: generator: { dtype: str } relations: gen_port: { key: generator, values: port } +variables: + gen_p: { dims: [snapshot, generator], bounds: { lower: 0 } } given: variables: flow: dims: [snapshot, port] description: what a port puts into its bus -variables: - gen_p: { dims: [snapshot, generator], bounds: { lower: 0 } } constraints: gen_injects: dims: [snapshot, generator] @@ -156,7 +156,7 @@ expressions: | `dims` | required. The dimensions the row family runs over | | | `description` | free text | default `null` | -There is no `expression` and no `sense`, because nothing here builds the row. +There is no `expression` and no `sense`. `dual(name)` is the only place a given row family may be named, and the frame gives the reported expression its dimensions. A name declared under both `constraints:` and `given: constraints:` is refused. diff --git a/pyproject.toml b/pyproject.toml index 7c8dccdd..7670f013 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -194,12 +194,13 @@ extend-exclude = ["CHANGELOG.md"] [tool.typos.default.extend-words] # Each of these is deliberate, and none is reachable by fixing a spelling. # -# `wher` and `generatr` are misspellings *on purpose*: they are the input to the -# did-you-mean suggester, so correcting them deletes the case. `missable` is the +# `wher`, `generatr` and `balnce` are misspellings *on purpose*: they are the +# input to the did-you-mean suggester, so correcting them deletes the case. `missable` is the # word the sentence wants — a coordinate that can go missing — and is not # "miscible". `unparseable` is the accepted variant and is the name of a test. wher = "wher" generatr = "generatr" +balnce = "balnce" missable = "missable" unparseable = "unparseable" # `NAMEs` in scripts/setup-release-app.sh is a placeholder plus a plural, in a diff --git a/src/math_spec/model.py b/src/math_spec/model.py index 15979c7a..874efae7 100644 --- a/src/math_spec/model.py +++ b/src/math_spec/model.py @@ -761,16 +761,16 @@ class Spec(_StrictBlock): relations: dict[str, RelationBlock] = {} parameters: dict[str, ParameterBlock] = {} variables: dict[str, VariableBlock] = {} + #: What this file reads and does not build (:class:`GivenBlock`): columns + #: under ``variables:``, row families under ``constraints:``. Empty in a + #: file that stands alone. + given: GivenBlock = GivenBlock() constraints: dict[str, ConstraintBlock] = {} objective: ObjectiveBlock | None = None expressions: dict[str, ExpressionBlock] = {} macros: dict[str, MacroBlock] = {} piecewise: dict[str, PiecewiseBlock] = {} sos: dict[str, SosBlock] = {} - #: What this file reads and does not build (:class:`GivenBlock`): columns - #: under ``variables:``, row families under ``constraints:``. Empty in a - #: file that stands alone. - given: GivenBlock = GivenBlock() def relations_of(self, dimension: str) -> dict[str, RelationBlock]: """The relations with a column over *dimension*, by name.""" diff --git a/src/math_spec/typesetting/__init__.py b/src/math_spec/typesetting/__init__.py index 0575fe39..1260c4d5 100644 --- a/src/math_spec/typesetting/__init__.py +++ b/src/math_spec/typesetting/__init__.py @@ -181,8 +181,9 @@ def typeset_declaration( Raises: ValueError: *fmt* names no format. LanguageError: A model that does not compile; it does not print. - SchemaError: *name* is declared as none of the three, or as two — a - constraint may share a variable's name; or a symbol table entry + SchemaError: *name* is declared as none of the three, as two — a + constraint may share a variable's name — or under ``given:``, which + prints in the legend rather than as a line; or a symbol table entry names nothing in the model. """ walk = _walk(model, fmt, symbols, inline_expressions=inline_expressions) @@ -190,6 +191,15 @@ def typeset_declaration( kinds = {'named expression': schema.expressions, 'constraint': schema.constraints, 'variable': schema.variables} found = [kind for kind, group in kinds.items() if name in group] if not found: + givens = {'variable': schema.given.variables, 'constraint': schema.given.constraints} + given_kind = next((kind for kind, group in givens.items() if name in group), None) + if given_kind is not None: + msg = ( + f"'{name}' is a given {given_kind}, and a given declaration prints no line of its own — " + f"this file reads it and does not build it. It prints in the legend, under 'Given', " + f'so call typeset() for the whole model.' + ) + raise SchemaError(msg) everything = {n for group in kinds.values() for n in group} msg = f"'{name}' is not a named expression, constraint or variable. {did_you_mean(name, everything)}" raise SchemaError(msg) diff --git a/src/math_spec/typesetting/symbols.py b/src/math_spec/typesetting/symbols.py index ae0bf154..85b13a8a 100644 --- a/src/math_spec/typesetting/symbols.py +++ b/src/math_spec/typesetting/symbols.py @@ -43,22 +43,22 @@ ) # fmt: skip -def _word(name: str, fmt: Format, *, given: bool) -> str: - r"""One name as one symbol: upright where *given*, italic where chosen. +def _word(name: str, fmt: Format, *, upright: bool) -> str: + r"""One name as one symbol: *upright* where the data supplies it, italic where chosen. A Greek name is set as the letter only where chosen. Upright lower-case Greek needs ``upgreek``, which the two-package preamble and GitHub's - MathJax both lack, so a given ``eta`` prints as ``\mathrm{eta}``; a table + MathJax both lack, so an upright ``eta`` prints as ``\mathrm{eta}``; a table entry is how an author who loads ``upgreek`` writes ``\upeta``. """ - if given: + if upright: return fmt.upright(name) if name in _GREEK: return fmt.greek(name) return name if len(name) == 1 else fmt.italic(name) -def _derive_name_symbol(name: str, declared: frozenset[str], fmt: Format, *, given: bool = False) -> str: +def _derive_name_symbol(name: str, declared: frozenset[str], fmt: Format, *, upright: bool = False) -> str: r"""``p`` → ``p``; ``load`` → ``\mathit{load}``; ``p_max`` → ``p^{\mathrm{max}}``. An underscore is a qualifier, landing in the superscript, only where its @@ -68,8 +68,8 @@ def _derive_name_symbol(name: str, declared: frozenset[str], fmt: Format, *, giv """ head, _, tail = name.partition('_') if tail and (len(head) == 1 or head in _GREEK or head in declared): - return fmt.superscript(_word(head, fmt, given=given), fmt.upright(tail.replace('_', ','))) - return _word(name, fmt, given=given) + return fmt.superscript(_word(head, fmt, upright=upright), fmt.upright(tail.replace('_', ','))) + return _word(name, fmt, upright=upright) def chosen_expressions(schema: _ExpandedSpec) -> frozenset[str]: @@ -117,7 +117,7 @@ def __init__(self, schema: _ExpandedSpec, fmt: Format, table: SymbolTable) -> No self.name: dict[str, str] = { name: table.names[name] if name in table.names - else _derive_name_symbol(name, declared, fmt, given=name not in chosen) + else _derive_name_symbol(name, declared, fmt, upright=name not in chosen) for name in names } spoken_for = {s for s in self.name.values() if len(s) == 1} @@ -128,7 +128,7 @@ def __init__(self, schema: _ExpandedSpec, fmt: Format, table: SymbolTable) -> No #: an entry in :attr:`name`. Given structure, so upright unless a table #: overrides it. self.constraint: dict[str, str] = { - name: table.names[name] if name in table.names else _derive_name_symbol(name, declared, fmt, given=True) + name: table.names[name] if name in table.names else _derive_name_symbol(name, declared, fmt, upright=True) for name in (*schema.constraints, *schema.given.constraints) } diff --git a/tests/typesetting/test_declaration.py b/tests/typesetting/test_declaration.py index d1872884..2a537f9e 100644 --- a/tests/typesetting/test_declaration.py +++ b/tests/typesetting/test_declaration.py @@ -6,7 +6,7 @@ from __future__ import annotations -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any import pytest @@ -124,16 +124,34 @@ def test_a_body_naming_another_expression_inlines_it_on_its_own_and_names_it_in_ ) +#: A column and a row family this file reads, each named by something that prints. +GIVEN = override( + PLAIN, + **{ + 'given.variables.flow': {'dims': ['snapshot']}, + 'given.constraints.clearing': {'dims': ['snapshot']}, + 'expressions.price': 'dual(clearing)', + 'expressions.drawn': 'flow * 2', + }, +) + + @pytest.mark.parametrize( - ('name', 'match'), + ('model', 'name', 'match'), [ - pytest.param('spent', r"'spent' is not a named expression, constraint or variable.*spend", id='a-near-miss'), - pytest.param('objective', r"'objective' is not a named expression", id='the-objective-has-no-name'), + pytest.param( + PLAIN, 'spent', r"'spent' is not a named expression, constraint or variable.*spend", id='a-near-miss' + ), + pytest.param(PLAIN, 'objective', r"'objective' is not a named expression", id='the-objective-has-no-name'), + pytest.param(GIVEN, 'flow', r"'flow' is a given variable.*no line of its own.*legend", id='a-given-variable'), + pytest.param( + GIVEN, 'clearing', r"'clearing' is a given constraint.*no line of its own.*legend", id='a-given-constraint' + ), ], ) -def test_a_name_declared_as_none_of_the_three_is_refused(name: str, match: str): +def test_a_name_that_prints_no_line_of_its_own_is_refused(model: dict[str, Any], name: str, match: str): with pytest.raises(SchemaError, match=match): - typeset_declaration(PLAIN, name, 'latex') + typeset_declaration(model, name, 'latex') def test_a_name_shared_by_a_constraint_and_a_variable_is_refused_rather_than_guessed(): diff --git a/tests/typesetting/test_walk.py b/tests/typesetting/test_walk.py index 1fa637a9..9f58968b 100644 --- a/tests/typesetting/test_walk.py +++ b/tests/typesetting/test_walk.py @@ -489,7 +489,7 @@ def test_a_given_quantity_is_upright(name: str, expected: str): r"""Upright is what the data supplies, and it admits no exception — not for a single letter, and not for a Greek name, where an italic `\eta` that might be either is worse than an upright `\mathrm{eta}` that is one.""" - assert _derive_name_symbol(name, frozenset({'p', 'soc'}), LATEX, given=True) == expected + assert _derive_name_symbol(name, frozenset({'p', 'soc'}), LATEX, upright=True) == expected @EVERY_FORMAT From bf9bfeb0ab7edb43a3178dad2537cec2c3ede5b5 Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 17:08:42 +0200 Subject: [PATCH 3/9] feat(language): a patch file extends a model rather than a copy of it override(base, patches) lays each patch over the base a field at a time and hands back one mapping for to_spec. A partial entry must land on a declaration the base has, and a miss is refused with the near miss named. A whole entry creates. null at declaration level removes, and a stale removal is refused. Sibling patches must write disjoint paths, so their order never decides a model. A dimension or a relation may be added or restated exactly and never changed. The objective is one declaration laid over by the same rules. given: is laid over one kind at a time, entry by entry. Base and patches are copied, never mutated. override is exported from math_spec. The tests' own fixture override is renamed varied so the verb owns the word. docs/howto/compose.md carries the recipe and the four refusals, each produced by running the case. --- docs/about/limits.md | 6 + docs/howto/compose.md | 135 ++++++++++ mkdocs.yml | 1 + src/math_spec/__init__.py | 2 + src/math_spec/composition.py | 352 ++++++++++++++++++++++++++ tests/fixtures.py | 10 +- tests/test_advice.py | 10 +- tests/test_boundedness.py | 4 +- tests/test_composition.py | 254 +++++++++++++++++++ tests/test_dimensions.py | 6 +- tests/test_lowering.py | 30 +-- tests/test_piecewise.py | 16 +- tests/test_public_surface.py | 2 + tests/test_validation.py | 22 +- tests/typesetting/test_cases.py | 14 +- tests/typesetting/test_declaration.py | 12 +- tests/typesetting/test_formats.py | 14 +- tests/typesetting/test_symbols.py | 6 +- tests/typesetting/test_walk.py | 40 ++- 19 files changed, 842 insertions(+), 94 deletions(-) create mode 100644 docs/howto/compose.md create mode 100644 src/math_spec/composition.py create mode 100644 tests/test_composition.py diff --git a/docs/about/limits.md b/docs/about/limits.md index b20b8ac0..d6df2788 100644 --- a/docs/about/limits.md +++ b/docs/about/limits.md @@ -156,3 +156,9 @@ a path, so a model assembled in Python is checked exactly as a file is, and `Spec.to_yaml()` writes the file a reviewer reads. A `dict` may hold only what a file may hold. A built-in merge, and namespaces so that two fragments can each declare a `p`, are both things a library does before it hands over a `dict`. + +A project that extends a model it does not own writes a patch, not a copy. +`override` lays the patch over the base a field at a time, and refuses a patch +that lands on nothing, two patches that write one field, and an axis redeclared +under the math. The recipe is in +[compose a model from several files](../howto/compose.md). diff --git a/docs/howto/compose.md b/docs/howto/compose.md new file mode 100644 index 00000000..b382983c --- /dev/null +++ b/docs/howto/compose.md @@ -0,0 +1,135 @@ + + +# Compose a model from several files + +Build one model out of files that each say part of it. `override` lays +**patches** over a **base**: the model a framework ships, and the change a +project makes to it. It hands back one mapping, which +[`to_spec`](../reference/language/errors.md#what-to_spec-checks) loads like any +file. + +## A base and its patches + +1. **Write the base as a model**, and each patch as the change it makes. A + patch names only the fields it changes. A declaration a patch does not name + stays as the base wrote it. + + ```yaml title="base.yaml" + dimensions: + snapshot: { dtype: int } + generator: { dtype: str } + parameters: + capacity: { dims: [generator] } + cost: { dims: [generator] } + load: { dims: [snapshot] } + variables: + dispatch: { dims: [snapshot, generator], bounds: { lower: 0, upper: capacity } } + constraints: + power_balance: + dims: [snapshot] + expression: sum(dispatch, over=generator) == load + objective: + sense: minimize + expression: sum(dispatch * cost) + ``` + + ```yaml title="operate.yaml" + variables: + dispatch: { where: "capacity > 0" } + ``` + + ```yaml title="carbon.yaml" + parameters: + emission_rate: { dims: [generator] } + constraints: + emission_cap: + dims: [] + expression: sum(dispatch * emission_rate) <= 1000 + ``` + +2. **Lay the patches on the base.** Each patch is given a name, and that name + is what a refusal calls it. The patches must write different fields, so the + order they are given in cannot change the model. + + ```python + import math_spec as ms + + model = ms.override('base.yaml', {'carbon': 'carbon.yaml', 'operate': 'operate.yaml'}) + spec = ms.to_spec(model) + ``` + + `spec` declares `emission_cap` beside `power_balance`, and `dispatch` carries + the mask `capacity > 0`. + +3. **Remove a declaration with `null`.** A patch that does not mention a + declaration leaves it alone, so removal needs a marker of its own. + + ```yaml title="feasibility.yaml" + constraints: + emission_cap: null + objective: null + ``` + + The marker is the declaration itself. Deeper down, `null` is a value the + schema takes: `dispatch: { where: null }` gives that variable no mask, and + leaves the variable in place. + +4. **Nest the calls where one patch refines another.** The second call lays + its patch on the first call's result, so the order is on the page. + + ```python + model = ms.override(ms.override('base.yaml', {'pathway': 'pathway.yaml'}), {'project': 'project.yaml'}) + ``` + +## What a patch may say + +| The entry | What happens | +| ------------------------------------ | ---------------------------------------------------------- | +| some fields of a declaration | those fields change, and the rest of the declaration stays | +| a whole declaration under a new name | it is added | +| `null` under a declaration's name | it is removed | +| a dimension or a relation | it is added, or restated exactly as the base declares it | +| an entry under `given: variables:` or `given: constraints:` | it is edited, added or removed like any declaration, and the other kind stays | +| `version`, `description` | the patch's value replaces the base's | + +## A partial entry on a missing name + +An entry naming some fields has to land on a declaration the base has. A +mistyped name is refused rather than read as a new declaration: + +```text +patch 'project' edits the constraint 'power_balnce', which its base does not declare. Did you mean 'power_balance'? A patch creates a declaration only by writing it whole, and this one is not: a constraint needs `expression`. +``` + +To add a constraint, write the whole constraint. To change one, spell its name +as the base spells it. + +## Two patches on one field + +Two patches writing one field is refused, both named: + +```text +patches 'pathway' and 'project': both write variables.dispatch.bounds.upper. Patches laid on one base are disjoint, so nothing decides which of two writes wins. Write the change in one patch, or lay one patch on the result of the other: override(override(base, {'pathway': …}), {'project': …}). +``` + +## An axis redeclared + +A patch may add a dimension or a relation, and may restate one the base +declares. Changing one under the expressions already written over it is +refused: + +```text +patch 'relabelled' declares the dimension 'snapshot' as {'dtype': 'str'}, where its base declares {'dtype': 'int'}. A patch adjusts the math, not the axes the math is already written over: restate the declaration exactly, leave it out, or give the patch an axis of its own under a name of its own. +``` + +## A stale removal + +A removal says what the base has, so a removal of a declaration the base does +not have is refused with the near miss: + +```text +patch 'stale' removes the constraint 'power_balnce', which its base does not declare. A removal is a claim about what is there, so a stale one is a patch that no longer describes the model it lands on. Did you mean 'power_balance'? +``` diff --git a/mkdocs.yml b/mkdocs.yml index ccc32aff..e7107826 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -41,6 +41,7 @@ nav: - Declare a column of data: howto/declare-a-column.md - Fix a quantity that is data in one model and a decision in another: howto/pin-a-variable.md - Write a piecewise curve out by hand: howto/curve-by-hand.md + - Compose a model from several files: howto/compose.md - Reference: - Language: - reference/language/index.md diff --git a/src/math_spec/__init__.py b/src/math_spec/__init__.py index af4b52d7..f281b21a 100644 --- a/src/math_spec/__init__.py +++ b/src/math_spec/__init__.py @@ -12,6 +12,7 @@ from math_spec import program from math_spec.advice import advice +from math_spec.composition import override from math_spec.errors import ( ADVICE_KINDS, Advice, @@ -74,6 +75,7 @@ 'call_shape_error', 'did_you_mean', 'edge_error', + 'override', 'program', 'schema_error', 'to_latex', diff --git a/src/math_spec/composition.py b/src/math_spec/composition.py new file mode 100644 index 00000000..db34d5c7 --- /dev/null +++ b/src/math_spec/composition.py @@ -0,0 +1,352 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +"""Several files into one model, before any of them is validated. + +:func:`override` lays **patches** over a **base**: what a framework ships and a +project extends. A patch says only what it changes, because declarations are +laid over a field at a time:: + + constraints: + ramp: {dims: [snapshot, generator, investment_period]} + +A patch is not a :class:`~math_spec.model.Spec`. It is read before validation, +so it may carry ``null`` where a declaration would go and may name what only +its base declares. Nothing here resolves a name or checks a dim: the laid +mapping goes through :func:`~math_spec.validation.to_spec` like any other file. + +What a patch may say, and what is refused: + +* **A partial entry edits, and a whole one creates.** An entry that does not + validate as a declaration on its own has to land on one the base declares, + and a miss is refused with the near miss named. +* **Sibling patches are disjoint.** Two patches writing one field is refused, + both named, so the order they are given in never decides a model. Layering + is written out as ``override(override(base, …), …)``. +* **A patch adjusts the math, not the axes.** A ``dimensions`` or ``relations`` + entry may be added or restated exactly, never changed. +* **A declaration set to** ``null`` **is removed**, and a removal of what the + base does not declare is refused. The marker is positional: ``constraints: + {ramp: null}`` removes the constraint, where ``variables: {p: {where: null}}`` + sets that variable's mask to none, which is a value the schema takes. +* **``given:`` is laid over one kind at a time**, by the same rules as any + owned section. +""" + +from __future__ import annotations + +from copy import deepcopy +from typing import TYPE_CHECKING, Any, cast, get_args + +from pydantic import BaseModel, ValidationError + +from math_spec._yaml import read_model +from math_spec.errors import LanguageError, did_you_mean, schema_error +from math_spec.model import GivenBlock, Spec + +if TYPE_CHECKING: + from collections.abc import Iterable, Mapping + from pathlib import Path + +#: The declarations that are the coordinate space rather than the math. A patch +#: may add one, and may restate one its base already declares; it may not say +#: something else about it. +SHARED_SECTIONS = ('dimensions', 'relations') + +#: The declarations a patch edits, creates or removes. +OWNED_SECTIONS = ('parameters', 'variables', 'constraints', 'expressions', 'macros', 'piecewise', 'sos') + +#: What ``given:`` holds, by the key each kind sits under and what one entry of it is called. +GIVEN_KINDS = {'variables': 'given variable', 'constraints': 'given constraint'} + +#: Every section keyed by declaration name. ``objective`` is one declaration +#: rather than a mapping of them, and is laid over field by field beside these. +SECTIONS = (*SHARED_SECTIONS, *OWNED_SECTIONS, 'given') + +#: What one entry is called where dropping the key's last letter does not say it. +IRREGULAR = { + 'piecewise': 'piecewise curve', + 'sos': 'special-ordered set', + 'objective': 'objective', +} + + +def override( + base: str | Path | dict[str, Any] | Spec, + patches: Mapping[str, str | Path | dict[str, Any] | Spec], +) -> dict[str, Any]: + """*base* with each patch laid over it, and nothing laid over another patch. + + Args: + base: The model being extended: a YAML path, YAML text, a mapping, or a + loaded :class:`~math_spec.model.Spec`. + patches: What each patch is called, to the patch. The name is what an + error calls it. The patches must write disjoint fields, so the + order they are given in cannot change the result. + + Returns: + One mapping, ready for :func:`~math_spec.validation.to_spec`. Nothing + in it has been resolved, name-checked or lowered, and it shares no + object with *base* or any patch. + + Raises: + LanguageError: A patch edits or removes a declaration its base does not + declare; a patch creates one that is not whole; a patch redeclares + a dimension or a relation as something else; or two patches write + one field. + FileNotFoundError: A ``str`` with no newline that names no file. + """ + read = {name: _declarations(patch) for name, patch in patches.items()} + _disjoint(read) + + result = deepcopy(_declarations(base)) + for name, patch in read.items(): + result = _lay_over(result, deepcopy(patch), name) + return result + + +def _declarations(source: str | Path | dict[str, Any] | Spec) -> dict[str, Any]: + """A base or a patch as the mapping it declares, whatever shape it arrived in. + + Deliberately not :func:`~math_spec.validation.to_spec`: a patch carrying a + ``null`` or naming only the field it changes is not a model, and validating + it here would refuse the files this module exists to read. + """ + if isinstance(source, Spec): + return source.to_dict() + if isinstance(source, dict): + return source + return read_model(source) + + +def _singular(section: str) -> str: + """What one entry in *section* is called, ``sos`` and ``piecewise`` not being plurals.""" + return IRREGULAR.get(section, section[:-1]) + + +def _entry_class(owner: type[BaseModel], field: str) -> type[BaseModel]: + """The schema's own class for one entry under *field* of *owner*. + + Read off the annotation rather than listed here, so a section added to the + schema cannot be laid over by a rule that does not know what it is made of. + """ + annotation = owner.model_fields[field].annotation + inner = [arg for arg in get_args(annotation) if arg is not type(None)] + return cast('type[BaseModel]', inner[-1] if inner else annotation) + + +def _whole(cls: type[BaseModel], block: Any) -> bool: + """Whether *block* is a declaration on its own, which is what lets a patch create one.""" + try: + cls.model_validate(block) + except ValidationError: + return False + return True + + +def _incomplete(label: str, cls: type[BaseModel], block: Any) -> str: + """What *block* is short of, in the schema's own words rather than a second list.""" + fields = cls.model_fields + missing = sorted(name for name, field in fields.items() if field.is_required() and name not in (block or {})) + if missing: + return f'{_a(label)} needs {_and_list(missing)}' + try: + cls.model_validate(block) + except ValidationError as e: + return str(schema_error(e)) + raise AssertionError(f'{_a(label)} asked what it is short of is whole: {block!r}') + + +def _a(noun: str) -> str: + """*noun* under the article that reads: an objective, a constraint.""" + return f'an {noun}' if noun[0] in 'aeiou' else f'a {noun}' + + +def _and_list(names: Iterable[str]) -> str: + """``a``, ``a and b``, ``a, b and c``: the field names a message ends on.""" + spelled = [f'`{name}`' for name in names] + if len(spelled) == 1: + return spelled[0] + return f'{", ".join(spelled[:-1])} and {spelled[-1]}' + + +def _writes(patch: Mapping[str, Any]) -> list[tuple[str, ...]]: + """Every field *patch* writes, as a path. + + A removal is the declaration's own path, so it overlaps every edit inside + that declaration: removing and editing one declaration is two patches + disagreeing, whichever order they would have been laid in. + """ + paths: list[tuple[str, ...]] = [] + for key, value in patch.items(): + if key in SECTIONS: + for name, block in (value or {}).items(): + paths.extend(_leaves((key, name), block)) + elif key == 'objective': + paths.extend(_leaves(('objective',), value)) + else: + paths.append((key,)) + return paths + + +def _leaves(prefix: tuple[str, ...], value: Any) -> list[tuple[str, ...]]: + """The paths *value* writes under *prefix*, a mapping being walked into and anything else a leaf.""" + if isinstance(value, dict) and value: + return [leaf for key, inner in value.items() for leaf in _leaves((*prefix, key), inner)] + return [prefix] + + +def _disjoint(read: Mapping[str, dict[str, Any]]) -> None: + """Refuse two patches that write one field, which is the only way order could matter.""" + claimed: dict[tuple[str, ...], str] = {} + for name, patch in read.items(): + for path in _writes(patch): + for other, owner in claimed.items(): + if path[: len(other)] == other or other[: len(path)] == path: + raise LanguageError(_overlap_message(owner, other, name, path)) + claimed[path] = name + + +def _overlap_message(owner: str, claimed: tuple[str, ...], name: str, path: tuple[str, ...]) -> str: + """The refusal for two patches writing one field, naming both and the rewrite.""" + where = f"'{owner}' writes {'.'.join(claimed)} and '{name}' writes {'.'.join(path)}" + if claimed == path: + where = f'both write {".".join(path)}' + return ( + f"patches '{owner}' and '{name}': {where}. Patches laid on one base are disjoint, so nothing " + f'decides which of two writes wins. Write the change in one patch, or lay one patch on the ' + f"result of the other: override(override(base, {{'{owner}': …}}), {{'{name}': …}})." + ) + + +def _lay_over(base: dict[str, Any], patch: dict[str, Any], name: str) -> dict[str, Any]: + """One patch over one base, a section at a time, the base left as it was.""" + laid = dict(base) + for key, value in patch.items(): + if key == 'given': + laid[key] = _given(laid.get(key) or {}, value or {}, name) + elif key in SHARED_SECTIONS: + laid[key] = _shared(laid.get(key) or {}, value or {}, key, name) + elif key in OWNED_SECTIONS: + laid[key] = _owned(laid.get(key) or {}, value or {}, _singular(key), _entry_class(Spec, key), name) + elif key == 'objective': + laid = _objective(laid, value, name) + else: + laid[key] = value + return laid + + +def _given(declared: dict[str, Any], patch: dict[str, Any], name: str) -> dict[str, Any]: + """The ``given:`` block, one kind laid over at a time, so naming the columns keeps the row families. + + A kind the block does not have is carried as written, and the closed + schema refuses it at load. + """ + out = dict(declared) + for kind, block in patch.items(): + if kind in GIVEN_KINDS: + cls = _entry_class(GivenBlock, kind) + out[kind] = _owned(out.get(kind) or {}, block or {}, GIVEN_KINDS[kind], cls, name) + else: + out[kind] = block + return out + + +def _shared(declared: dict[str, Any], patch: dict[str, Any], section: str, name: str) -> dict[str, Any]: + """One ``dimensions`` or ``relations`` block: a patch adds an axis or restates one, never changes it.""" + out = dict(declared) + for key, block in patch.items(): + if block is None: + _removed(out, key, _singular(section), name) + elif key not in out: + out[key] = block + elif not _agrees(out[key], block): + raise LanguageError( + f"patch '{name}' declares the {_singular(section)} '{key}' as {block!r}, where its base " + f'declares {out[key]!r}. A patch adjusts the math, not the axes the math is already ' + f'written over: restate the declaration exactly, leave it out, or give the patch an ' + f'axis of its own under a name of its own.' + ) + return out + + +def _agrees(under: Any, over: Any) -> bool: + """Whether *over* says only what *under* already says, a field the patch omits being no claim.""" + if isinstance(over, dict): + return isinstance(under, dict) and all( + key in under and _agrees(under[key], value) for key, value in over.items() + ) + return bool(under == over) + + +def _owned( + declared: dict[str, Any], patch: dict[str, Any], label: str, cls: type[BaseModel], name: str +) -> dict[str, Any]: + """One section of the math, each entry editing what is there or creating what is whole.""" + out = dict(declared) + for key, block in patch.items(): + if block is None: + _removed(out, key, label, name) + elif key in out: + out[key] = _field_by_field(out[key], block) + elif _whole(cls, block): + out[key] = block + else: + raise LanguageError( + f"patch '{name}' edits the {label} '{key}', which its base does not declare. " + f'{did_you_mean(key, list(out))} A patch creates a declaration only by writing it whole, ' + f'and this one is not: {_incomplete(label, cls, block)}.' + ) + return out + + +def _removed(out: dict[str, Any], key: str, label: str, name: str) -> None: + """Delete what the patch nulled, refusing a removal its base cannot satisfy.""" + if key not in out: + raise LanguageError( + f"patch '{name}' removes the {label} '{key}', which its base does not declare. " + f'A removal is a claim about what is there, so a stale one is a patch that no longer describes ' + f'the model it lands on. ' + did_you_mean(key, list(out)) + ) + del out[key] + + +def _objective(laid: dict[str, Any], patch: Any, name: str) -> dict[str, Any]: + """The one declaration that is not keyed by a name, laid over by the same three rules.""" + out = dict(laid) + standing = out.get('objective') + cls = _entry_class(Spec, 'objective') + if patch is None: + if standing is None: + raise LanguageError( + f"patch '{name}' removes the objective, which its base does not declare. A removal is a " + f'claim about what is there, and a model with no objective is already the feasibility ' + f'problem this patch is asking for.' + ) + del out['objective'] + elif standing is not None: + out['objective'] = _field_by_field(standing, patch) + elif _whole(cls, patch): + out['objective'] = patch + else: + raise LanguageError( + f"patch '{name}' edits the objective, which its base does not declare. A patch creates the " + f'objective only by writing it whole, and this one is not: {_incomplete("objective", cls, patch)}.' + ) + return out + + +def _field_by_field(under: Any, over: Any) -> Any: + """*over* laid on *under*: mappings merge, everything else replaces. + + ``None`` replaces here rather than removing. Removal is the + declaration-level marker and reaches no deeper, so ``where: null`` is the + mask the schema already lets a file write. + """ + if isinstance(under, dict) and isinstance(over, dict): + merged = dict(under) + for key, value in over.items(): + merged[key] = _field_by_field(merged.get(key), value) + return merged + return over diff --git a/tests/fixtures.py b/tests/fixtures.py index dc3cd0e0..30345aa6 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -23,7 +23,7 @@ OPERATOR_PROBES = sorted((EXAMPLES / 'operators').glob('*.yaml')) #: The shape of ``examples/dispatch.yaml`` as a dict a test can vary with -#: :func:`override`: no ``where:``, the constraint named ``balance``, and short +#: :func:`varied`: no ``where:``, the constraint named ``balance``, and short #: names, so a test that prints it asserts on the math rather than on the #: example's own vocabulary. DISPATCH_MODEL: dict[str, Any] = { @@ -55,10 +55,10 @@ } -def override(base: dict[str, Any], **patch: Any) -> dict[str, Any]: +def varied(base: dict[str, Any], **patch: Any) -> dict[str, Any]: """A deep copy of ``base`` with dotted paths replaced, missing parents created. - ``override(DISPATCH_MODEL, **{'variables.p.where': 'p_max > 0'})``. + ``varied(DISPATCH_MODEL, **{'variables.p.where': 'p_max > 0'})``. """ raw = copy.deepcopy(base) for dotted, value in patch.items(): @@ -71,12 +71,12 @@ def override(base: dict[str, Any], **patch: Any) -> dict[str, Any]: def schema_of(source: str | Path | dict[str, Any], **patch: Any) -> Spec: - """A ``Spec`` from a YAML path, YAML text, or a raw dict, ``**patch`` applied by :func:`override`. + """A ``Spec`` from a YAML path, YAML text, or a raw dict, ``**patch`` applied by :func:`varied`. ``Path`` means a file, ``str`` means the YAML itself. """ raw = raw_of(source) - return to_spec(override(raw, **patch) if patch else raw) + return to_spec(varied(raw, **patch) if patch else raw) def raw_of(source: str | Path | dict[str, Any]) -> dict[str, Any]: diff --git a/tests/test_advice.py b/tests/test_advice.py index 17f46f25..1a5e4d64 100644 --- a/tests/test_advice.py +++ b/tests/test_advice.py @@ -17,20 +17,20 @@ import pytest from math_spec import ADVICE_KINDS, advice, to_program, to_spec -from tests.fixtures import SMALL_MODEL, override +from tests.fixtures import SMALL_MODEL, varied if TYPE_CHECKING: from pathlib import Path #: ``h`` is the target of ``lk`` and nothing else reaches it; ``g`` is an axis. -TARGET_ONLY = override( +TARGET_ONLY = varied( SMALL_MODEL, variables={'p': {'dims': ['g']}}, objective={'sense': 'minimize', 'expression': 'sum(p * c)'}, ) #: The same with the relation gone, so nothing reaches ``h`` at all. -UNREACHED = override(TARGET_ONLY, relations={}) +UNREACHED = varied(TARGET_ONLY, relations={}) def test_a_dimension_nothing_reaches_is_named(): @@ -51,14 +51,14 @@ def test_a_dimension_nothing_reaches_is_named(): ], ) def test_a_dimension_something_reaches_is_in_use(patch): - assert not advice(override(TARGET_ONLY, **patch)), ( + assert not advice(varied(TARGET_ONLY, **patch)), ( 'a dimension a relation targets, a declaration indexes or a grouping lands on is in use' ) #: A model with one note of each kind: nothing reaches `h`, and `p` is driven #: down by the objective with an open lower bound and no constraint on it. -BOTH_KINDS = override(UNREACHED, **{'objective.expression': 'sum(p)', 'variables.p.bounds': {'lower': -float('inf')}}) +BOTH_KINDS = varied(UNREACHED, **{'objective.expression': 'sum(p)', 'variables.p.bounds': {'lower': -float('inf')}}) def test_both_kinds_of_note_come_through_the_one_door(): diff --git a/tests/test_boundedness.py b/tests/test_boundedness.py index b606dc73..ba5e2332 100644 --- a/tests/test_boundedness.py +++ b/tests/test_boundedness.py @@ -15,9 +15,9 @@ from math_spec.boundedness import unbounded_notes from math_spec.lowering import to_program from math_spec.operators import BUILTIN_NAMES -from tests.fixtures import SMALL_MODEL, override, schema_of +from tests.fixtures import SMALL_MODEL, schema_of, varied -BASE = override( +BASE = varied( SMALL_MODEL, variables={'v': {'dims': ['g']}, 'w': {'dims': ['g']}}, objective={'sense': 'minimize', 'expression': 'sum(v, over=g)'}, diff --git a/tests/test_composition.py b/tests/test_composition.py new file mode 100644 index 00000000..e1e840e7 --- /dev/null +++ b/tests/test_composition.py @@ -0,0 +1,254 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +"""`override` lays a base and its patches, and what it refuses. + +A name the patch declares is the point, and what is pinned for it is the +opposite: every collision the caller did not ask for is an error naming both +sides. A patch that lands on nothing, two patches writing one field, and an +axis redeclared under the expressions written over it are the three, and each +one is a model that would otherwise load and mean something nobody wrote. +""" + +from __future__ import annotations + +import copy + +import pytest + +from math_spec import LanguageError, override, to_markdown, to_spec +from tests.fixtures import DISPATCH_MODEL + +#: A patch that adds what it needs and a constraint that reads it, so the +#: composed model is one `to_spec` accepts rather than only one that lays. +CARBON = { + 'parameters': {'co2': {'dims': ['generator']}}, + 'constraints': {'co2_cap': {'dims': [], 'expression': 'sum(p * co2) <= 100'}}, +} + +#: `DISPATCH_MODEL` with no objective, for the patches that ask about one. +FEASIBILITY = {k: v for k, v in DISPATCH_MODEL.items() if k != 'objective'} + + +def test_a_patch_names_only_the_field_it_changes(): + laid = override(DISPATCH_MODEL, {'operate': {'variables': {'p': {'where': 'p_max > 0'}}}}) + assert laid['variables']['p'] == { + 'dims': ['snapshot', 'generator'], + 'bounds': {'lower': 0, 'upper': 'p_max'}, + 'where': 'p_max > 0', + }, 'the fields the patch does not name are the ones the base declared' + + +def test_the_base_and_the_patches_are_never_mutated(): + """The guarantee belongs to the function rather than to the caller's discipline.""" + patches = {'carbon': CARBON, 'operate': {'variables': {'p': {'where': 'p_max > 0'}}}} + before = copy.deepcopy((DISPATCH_MODEL, patches)) + override(DISPATCH_MODEL, patches) + assert before == (DISPATCH_MODEL, patches), 'a patched base is a new mapping, and both inputs are untouched' + + +def test_a_whole_declaration_is_created_and_the_model_loads(): + spec = to_spec(override(DISPATCH_MODEL, {'carbon': CARBON})) + assert 'co2_cap' in spec.constraints + assert to_markdown(spec), 'a composed model is one a reviewer can read as math' + + +@pytest.mark.parametrize( + ('patch', 'says'), + [ + pytest.param({'constraints': {'balnce': {'dims': ['snapshot']}}}, "Did you mean 'balance'?", id='a-near-miss'), + pytest.param( + {'constraints': {'co2_cap': {'dims': []}}}, 'a constraint needs `expression`', id='short-of-a-field' + ), + pytest.param({'parameters': {'co2': {'dtype': 'float'}}}, 'a parameter needs `dims`', id='short-of-its-frame'), + pytest.param( + {'expressions': {'spend': {'dims': ['snapshot']}}}, + 'one `expression:` or a set of `cases:`', + id='short-of-what-it-says', + ), + pytest.param( + {'given': {'variables': {'flow': {'domain': 'binary'}}}}, + 'a given variable needs `dims`', + id='a-given-column-short-of-its-frame', + ), + ], +) +def test_a_partial_entry_that_lands_on_nothing_is_refused(patch, says): + """The typo case: laying a partial entry on nothing would invent a declaration nothing refers to.""" + with pytest.raises(LanguageError, match=r'does not declare') as raised: + override(DISPATCH_MODEL, {'project': patch}) + assert says in str(raised.value), 'the refusal says what the entry is short of, or what it nearly named' + + +def test_a_null_removes_a_declaration_and_the_model_still_loads(): + laid = override(DISPATCH_MODEL, {'unconstrained': {'constraints': {'balance': None}}}) + assert laid['constraints'] == {}, 'the declaration is gone rather than emptied' + assert to_spec(laid).constraints == {} + + +def test_a_stale_removal_is_refused(): + with pytest.raises(LanguageError, match=r"'balnce'.*does not declare.*Did you mean 'balance'\?"): + override(DISPATCH_MODEL, {'stale': {'constraints': {'balnce': None}}}) + + +def test_a_null_inside_a_declaration_is_a_value_rather_than_a_removal(): + """`where: null` is the mask the schema already takes, so the marker is positional. + + The base carries a mask, so setting the field to `null` and deleting it are + two different declarations rather than the same one twice. + """ + masked = override(DISPATCH_MODEL, {'masked': {'variables': {'p': {'where': 'p_max > 0'}}}}) + laid = override(masked, {'unmasked': {'variables': {'p': {'where': None}}}}) + assert 'where' in laid['variables']['p'], 'the field is set to none, and is not deleted from the declaration' + assert laid['variables']['p']['where'] is None + assert to_spec(laid).variables['p'].where is None + + +def test_a_null_two_levels_down_is_a_value_too(): + """The removal marker reaches no deeper than the declaration, however deep the `null` sits.""" + laid = override(DISPATCH_MODEL, {'unbounded': {'variables': {'p': {'bounds': {'upper': None}}}}}) + assert laid['variables']['p']['bounds'] == {'lower': 0, 'upper': None}, ( + 'the bound is set to none beside the one the base keeps, and neither is deleted' + ) + + +def test_the_result_shares_no_declaration_with_the_base_or_the_patch(): + """Both sides are copied, so editing a composed model cannot reach back into either. + + A declaration no patch names is the case worth pinning: it is carried over + untouched, which is exactly where a reference would be passed on instead. + """ + laid = override(DISPATCH_MODEL, {'carbon': CARBON}) + assert laid['parameters']['load'] is not DISPATCH_MODEL['parameters']['load'], ( + "a declaration the patches leave alone is a copy, not the base's own object" + ) + assert laid['parameters']['co2'] is not CARBON['parameters']['co2'], ( + "a declaration a patch adds is a copy, not the patch mapping's own object" + ) + + +@pytest.mark.parametrize( + 'patches', + [ + pytest.param( + { + 'pathway': {'variables': {'p': {'bounds': {'upper': 'p_max'}}}}, + 'project': {'variables': {'p': {'bounds': {'upper': 'cost'}}}}, + }, + id='one-field-twice', + ), + pytest.param( + { + 'pathway': {'constraints': {'balance': None}}, + 'project': {'constraints': {'balance': {'dims': ['snapshot', 'generator']}}}, + }, + id='removed-here-edited-there', + ), + pytest.param( + {'pathway': {'objective': {'sense': 'maximize'}}, 'project': {'objective': {'sense': 'minimize'}}}, + id='the-objective-twice', + ), + pytest.param( + {'pathway': {'version': 0}, 'project': {'version': 1}}, + id='a-top-level-scalar-twice', + ), + ], +) +def test_two_patches_that_write_one_field_are_refused(patches): + with pytest.raises(LanguageError) as raised: + override(DISPATCH_MODEL, patches) + message = str(raised.value) + assert "'pathway'" in message and "'project'" in message, 'a collision names both patches, not just the second' + assert 'override(override(' in message, 'the message names the rewrite, which is to lay one on the other' + + +def test_disjoint_patches_compose_the_same_model_in_either_order(): + """What the disjointness rule buys: the argument's position never decides a model.""" + patches = {'carbon': CARBON, 'operate': {'variables': {'p': {'where': 'p_max > 0'}}}} + reversed_order = dict(reversed(list(patches.items()))) + assert override(DISPATCH_MODEL, patches) == override(DISPATCH_MODEL, reversed_order) + + +def test_layering_is_written_out_as_nesting(): + """The second call lays on the first's result, which is where an order is allowed to matter.""" + once = override(DISPATCH_MODEL, {'pathway': {'variables': {'p': {'where': 'p_max > 0'}}}}) + twice = override(once, {'project': {'variables': {'p': {'where': 'cost > 0'}}}}) + assert twice['variables']['p']['where'] == 'cost > 0' + + +def test_a_patch_adds_an_axis_and_may_restate_one_it_shares(): + laid = override( + DISPATCH_MODEL, + {'periods': {'dimensions': {'snapshot': {'dtype': 'int'}, 'investment_period': {'dtype': 'int'}}}}, + ) + assert sorted(laid['dimensions']) == ['generator', 'investment_period', 'snapshot'], ( + 'the axis the patch adds joins the two the base declares, and the restated one is not doubled' + ) + + +def test_a_patch_that_redeclares_an_axis_is_refused(): + """An axis changed under the expressions already written over it is a different model, silently.""" + with pytest.raises(LanguageError, match=r'adjusts the math, not the axes'): + override(DISPATCH_MODEL, {'relabelled': {'dimensions': {'snapshot': {'dtype': 'str'}}}}) + + +def test_the_objective_is_laid_over_field_by_field(): + laid = override(DISPATCH_MODEL, {'maximised': {'objective': {'sense': 'maximize'}}}) + assert laid['objective'] == {'sense': 'maximize', 'expression': 'sum(p * cost)'}, ( + 'the sense the patch names changes, and the expression the base wrote stays' + ) + + +def test_the_objective_can_be_removed_and_the_model_is_a_feasibility_problem(): + laid = override(DISPATCH_MODEL, {'feasible': {'objective': None}}) + assert 'objective' not in laid + assert to_spec(laid).objective is None + + +def test_a_whole_objective_is_created_where_the_base_has_none(): + laid = override(FEASIBILITY, {'priced': {'objective': DISPATCH_MODEL['objective']}}) + assert to_spec(laid).objective is not None + + +@pytest.mark.parametrize( + ('patch', 'says'), + [ + pytest.param({'objective': None}, 'already the feasibility problem', id='removing-one-that-is-not-there'), + pytest.param( + {'objective': {'sense': 'maximize'}}, 'an objective needs `expression`', id='editing-one-that-is-not-there' + ), + ], +) +def test_an_objective_a_base_does_not_declare_is_refused(patch, says): + with pytest.raises(LanguageError, match=r'does not declare') as raised: + override(FEASIBILITY, {'project': patch}) + assert says in str(raised.value) + + +def test_a_patch_over_one_kind_of_given_leaves_the_other_alone(): + """`given:` is laid over a kind at a time, so patching the columns cannot drop the row families.""" + base = { + 'dimensions': {'g': {'dtype': 'str'}}, + 'given': {'variables': {'p': {'dims': ['g']}}, 'constraints': {'cap': {'dims': ['g']}}}, + 'expressions': {'price': {'expression': 'dual(cap)'}}, + } + laid = override(base, {'wider': {'given': {'variables': {'p': {'domain': 'binary'}}}}}) + assert laid['given']['variables']['p'] == {'dims': ['g'], 'domain': 'binary'}, ( + 'the given column is edited field by field like any declaration' + ) + assert sorted(laid['given']['constraints']) == ['cap'], 'the kind the patch did not name is still there' + assert to_spec(laid).given.variables['p'].domain == 'binary' + + +def test_a_patch_is_a_path_as_readily_as_a_mapping(tmp_path): + """Whatever every other verb takes, so a patch travels as a file rather than as a script.""" + patch = tmp_path / 'carbon.yaml' + patch.write_text('parameters:\n co2: {dims: [generator]}\n', encoding='utf-8') + laid = override(DISPATCH_MODEL, {'carbon': str(patch)}) + assert 'co2' in laid['parameters'] + + +def test_a_loaded_spec_is_a_base_as_readily_as_a_mapping(): + laid = override(to_spec(DISPATCH_MODEL), {'carbon': CARBON}) + assert 'co2_cap' in to_spec(laid).constraints diff --git a/tests/test_dimensions.py b/tests/test_dimensions.py index b39a0d18..ca7b3e4a 100644 --- a/tests/test_dimensions.py +++ b/tests/test_dimensions.py @@ -14,7 +14,7 @@ from math_spec.program import Mask, RelationPairComparisonNode from math_spec.resolution import Namespace, expression_of, where_of from math_spec.validation import to_spec -from tests.fixtures import override, schema_of +from tests.fixtures import schema_of, varied if TYPE_CHECKING: from math_spec.model import Spec @@ -355,7 +355,7 @@ class TestTheEdgeRulesAreDecidedAtLoad: } def _refused(self, expression: str) -> str: - raw = override(self.BASE, **{'constraints.k.expression': expression}) + raw = varied(self.BASE, **{'constraints.k.expression': expression}) with pytest.raises(DimensionError) as caught: to_spec(raw) return str(caught.value) @@ -409,7 +409,7 @@ def test_a_zero_step_vacates_nothing_and_needs_no_edge(self): literal zero vacates none, so there is nothing for an `edge=` to answer for. A *named* offset may be zero in the data and is not known here. """ - to_spec(override(self.BASE, **{'constraints.k.expression': 'p <= shift(cap, along=g, offset=0)'})) + to_spec(varied(self.BASE, **{'constraints.k.expression': 'p <= shift(cap, along=g, offset=0)'})) # --------------------------------------------------------------------------- diff --git a/tests/test_lowering.py b/tests/test_lowering.py index 19a3f444..08623d1d 100644 --- a/tests/test_lowering.py +++ b/tests/test_lowering.py @@ -63,7 +63,7 @@ where_children, ) from math_spec.resolution import Namespace, expression_of, where_of -from tests.fixtures import DISPATCH_MODEL, EXAMPLES, SMALL_MODEL, override, schema_of +from tests.fixtures import DISPATCH_MODEL, EXAMPLES, SMALL_MODEL, schema_of, varied if TYPE_CHECKING: from math_spec._expression_parser import ArithmeticNode @@ -75,7 +75,7 @@ #: One dimension, one parameter, one bounded variable and a scalar constraint: #: the smallest model that loads, for a claim about the plan's record rather -#: than about the math in it. A test adds what it judges with :func:`override`. +#: than about the math in it. A test adds what it judges with :func:`varied`. TINY = { 'dimensions': {'g': {}}, 'parameters': {'cost': {'dims': ['g']}}, @@ -93,7 +93,7 @@ #: offset. Which node a construct becomes is mostly a claim about the dim it #: consumes and the dim it lands on, and stating that needs a third dimension #: and two relations over one of them. -SHAPES_MODEL = override( +SHAPES_MODEL = varied( SMALL_MODEL, **{ 'dimensions.z': {'dtype': 'str'}, @@ -157,7 +157,7 @@ def test_lower_program_structure(dispatch_program): @pytest.mark.parametrize('sense', [pytest.param('minimize', id='minimize'), pytest.param('maximize', id='maximize')]) def test_the_objective_sense_crosses_untranslated(sense: str): """One spelling from the file to the program, in both directions — each sink translates at its own edge.""" - program = to_program(override(TINY, objective={'sense': sense, 'expression': 'sum(p * cost, over=g)'})) + program = to_program(varied(TINY, objective={'sense': sense, 'expression': 'sum(p * cost, over=g)'})) assert program.objective is not None assert program.objective.sense == sense, "the file's own word for the direction, unchanged" @@ -298,7 +298,7 @@ def test_a_lowered_where_is_a_mask_that_answers_from_its_root(dispatch_program): def test_a_lowered_mask_answers_its_dims_conjuncts_and_atoms(variable, where, dims, conjuncts, atoms): """`Mask.dims` is read off the leaves, which carry their declarations' dims; `atoms` crosses the `OR` that `conjuncts` stops at.""" - mask = to_program(override(SMALL_MODEL, **{f'variables.{variable}.where': where})).variables[variable].where + mask = to_program(varied(SMALL_MODEL, **{f'variables.{variable}.where': where})).variables[variable].where assert mask.dims == frozenset(dims) assert len(mask.conjuncts) == conjuncts, 'an OR is one conjunct, a leaf is one conjunct' @@ -394,7 +394,7 @@ def test_an_unwritten_where_lowers_to_none_not_an_empty_mask(): def test_a_constraint_where_is_a_mask_like_a_variable_s(): - lowered = to_program(override(DISPATCH_MODEL, **{'constraints.balance.where': 'load > 0'})) + lowered = to_program(varied(DISPATCH_MODEL, **{'constraints.balance.where': 'load > 0'})) (c,) = lowered.constraints.values() assert c.where == Mask(ParameterComparisonNode('load', '>', 0.0, ('snapshot',))) @@ -714,7 +714,7 @@ def test_a_relation_names_the_dimension_its_values_label(): def test_an_unknown_dimension_is_a_near_miss_rather_than_an_empty_declaration(): """A typo used to return an empty declaration, silently dropping every join.""" - program = to_program(override(TINY, **{'dimensions.snapshot': {}})) + program = to_program(varied(TINY, **{'dimensions.snapshot': {}})) assert program.dimension('snapshot').dtype == 'str', 'a declared dimension still comes back' with pytest.raises(KeyError, match='snapshto') as excinfo: @@ -738,7 +738,7 @@ def test_a_program_seals_its_declaration_groups(dispatch_program, group): def test_expressions_are_the_ones_a_row_is_built_from(): """`expressions` named the *declared* ones, which build no row at all.""" program = to_program( - override( + varied( TINY, expressions={'spend': 'sum(cost, over=g)'}, objective={'sense': 'minimize', 'expression': 'sum(p * cost, over=g)'}, @@ -759,7 +759,7 @@ def test_expressions_are_the_ones_a_row_is_built_from(): def _footprint_of(constraint: str, objective: str) -> Footprint: return to_program( - override( + varied( TINY, constraints={'k': {'dims': ['g'], 'expression': constraint}}, objective={'sense': 'minimize', 'expression': objective}, @@ -809,7 +809,7 @@ def test_the_footprint_is_walked_once_and_held(dispatch_program): def test_a_named_expression_is_not_in_the_footprint(): """It builds no row, so counting it would answer wrongly about what is solved.""" - program = to_program(override(TINY, expressions={'spend': 'sum(p * cost, over=g)'})) + program = to_program(varied(TINY, expressions={'spend': 'sum(p * cost, over=g)'})) assert Parameter not in program.footprint.shapes, "the named expression's parameter reaches no row" assert Parameter in {type(n) for n in walk(program.named_expressions['spend'].expression)}, ( @@ -823,7 +823,7 @@ def test_a_dimension_carries_the_dtype_its_labels_are_checked_against(): A dimension is read from whatever table carries it, so nothing downstream can infer what the column should have been. """ - program = to_program(override(TINY, **{'dimensions.t': {'dtype': 'int'}})) + program = to_program(varied(TINY, **{'dimensions.t': {'dtype': 'int'}})) assert program.dimension('t').dtype == 'int', 'a declared dtype reaches the plan' assert program.dimension('g').dtype == 'str', "and the schema's default does too, rather than nothing" @@ -949,14 +949,14 @@ def test_a_cased_expression_is_readable_by_the_name_the_file_wrote(): ) def test_an_entry_is_in_the_math_where_the_objective_or_a_constraint_inlines_it(patch, in_math): """`in_math` is usage, not shape: one affine body is in the math when a row inlines it, however indirectly, and a reported quantity when none does.""" - program = to_program(override(TINY, expressions={'spend': 'sum(p * cost, over=g)'}, **patch)) + program = to_program(varied(TINY, expressions={'spend': 'sum(p * cost, over=g)'}, **patch)) assert program.named_expressions['spend'].in_math is in_math def test_an_entry_reached_only_through_another_is_in_the_math_with_it(): """The whole chain is in the math, not only the entry a row names: the constraint inlines `twice`, and `twice` inlines `spend`.""" program = to_program( - override( + varied( TINY, expressions={'spend': 'sum(p * cost, over=g)', 'twice': 'spend * 2'}, **{'constraints.c.expression': 'twice >= 1'}, @@ -971,7 +971,7 @@ def test_an_entry_reached_only_through_another_is_in_the_math_with_it(): def test_a_macro_formal_named_like_an_entry_keeps_the_entry_out_of_the_math(): """A formal shadows the entry inside the template, so the row inlines the argument, not the same-named entry.""" program = to_program( - override( + varied( TINY, expressions={'spend': 'sum(p * cost, over=g)'}, macros={'scaled': {'args': ['spend'], 'template': 'spend * 2'}}, @@ -985,7 +985,7 @@ def test_a_macro_formal_named_like_an_entry_keeps_the_entry_out_of_the_math(): def test_an_entry_that_reads_a_dual_is_a_reported_quantity(): """A dual is read after the solve, so an entry calling one is never in the math: it lowers to a Dual leaf and stays reported.""" - program = to_program(override(TINY, expressions={'shadow_price': 'dual(c)'})) + program = to_program(varied(TINY, expressions={'shadow_price': 'dual(c)'})) declaration = program.named_expressions['shadow_price'] assert declaration.in_math is False, 'the entry reading a dual is reported, never in the math' assert isinstance(declaration.expression, Dual), 'and it lowers to a Dual leaf' diff --git a/tests/test_piecewise.py b/tests/test_piecewise.py index 48d71537..a3fd7f81 100644 --- a/tests/test_piecewise.py +++ b/tests/test_piecewise.py @@ -31,7 +31,7 @@ MaskOf, check_message, ) -from tests.fixtures import DISPATCH_MODEL, override, raw_of, schema_of +from tests.fixtures import DISPATCH_MODEL, raw_of, schema_of, varied #: Larger than a minimal probe on purpose: a curve that exercises adjacency #: binaries and links is not something a smaller one can stand in for. @@ -69,12 +69,12 @@ sense: minimize expression: sum(op_cost, over=snapshot) """ -GATED = override( +GATED = varied( raw_of(NONCONVEX_YAML), **{'variables.u': {'dims': ['snapshot'], 'domain': 'binary'}, 'piecewise.cost_curve.activity': 'u'}, ) #: The convex curve stated as its segment lines, plus a binary the method cannot gate on. -LP = override( +LP = varied( raw_of(NONCONVEX_YAML), **{ 'piecewise.cost_curve.method': 'lp', @@ -83,9 +83,9 @@ }, ) #: The ``lp`` curve masked by one of its own values-parameters, so every check a block can carry is on it. -LP_MASKED = override(LP, **{'piecewise.cost_curve.points': 'bp_x'}) +LP_MASKED = varied(LP, **{'piecewise.cost_curve.points': 'bp_x'}) #: Two dims in the frame, so the emitted ``dims`` has an order to get wrong. -TWO_DIM = override( +TWO_DIM = varied( raw_of(NONCONVEX_YAML), **{ 'dimensions.generator': {'dtype': 'str'}, @@ -356,14 +356,14 @@ def test_a_gate_that_is_not_a_variable_is_refused(activity, match): #: ``lp`` bounded the other way: the same curve read as its lower envelope. -LP_CONCAVE = override( +LP_CONCAVE = varied( raw_of(NONCONVEX_YAML), **{ 'piecewise.cost_curve.method': 'lp', 'piecewise.cost_curve.links': [['p', 'bp_x'], ['op_cost', 'bp_y', '<=']], }, ) -CONVEX = override(raw_of(NONCONVEX_YAML), **{'piecewise.cost_curve.method': 'convex'}) +CONVEX = varied(raw_of(NONCONVEX_YAML), **{'piecewise.cost_curve.method': 'convex'}) #: Named so the completeness check below can read the answers back off them. @@ -419,7 +419,7 @@ def test_an_emitted_parameter_says_how_it_is_filled(): def test_a_file_supplied_mask_derives_nothing(): """A ``points:`` naming a parameter the file declared is bound like any other, and the mask check still names it.""" program = to_program( - override(LP, **{'parameters.reach': {'dims': ['bp'], 'dtype': 'bool'}, 'piecewise.cost_curve.points': 'reach'}) + varied(LP, **{'parameters.reach': {'dims': ['bp'], 'dtype': 'bool'}, 'piecewise.cost_curve.points': 'reach'}) ) assert program.parameters['reach'].derivation is None, 'the file declared it, so the caller binds it' diff --git a/tests/test_public_surface.py b/tests/test_public_surface.py index 5a1ff4de..03e4356b 100644 --- a/tests/test_public_surface.py +++ b/tests/test_public_surface.py @@ -25,6 +25,8 @@ { # the two public states, and the conversion to each 'Spec', 'to_spec', 'program', 'to_program', + # the file-level verb that lays patches over a base + 'override', # the error tree 'MathSpecError', 'LanguageError', 'SchemaError', 'DimensionError', 'PiecewiseExpansionError', 'did_you_mean', 'schema_error', diff --git a/tests/test_validation.py b/tests/test_validation.py index 5724a311..95ef07ee 100644 --- a/tests/test_validation.py +++ b/tests/test_validation.py @@ -19,20 +19,20 @@ from math_spec.resolution import Namespace, where_of from math_spec.typesetting import to_markdown from math_spec.validation import to_spec -from tests.fixtures import DISPATCH_MODEL, OPERATOR_PROBES, SMALL_MODEL, override +from tests.fixtures import DISPATCH_MODEL, OPERATOR_PROBES, SMALL_MODEL, varied if TYPE_CHECKING: from math_spec.model import Spec def _schema(**patch) -> Spec: - return to_spec(override(SMALL_MODEL, **patch)) + return to_spec(varied(SMALL_MODEL, **patch)) def _refusal(model: dict[str, Any] = SMALL_MODEL, **patch: Any) -> str: """The message `to_spec` refuses *model* patched with — and it has to refuse.""" with pytest.raises(LanguageError) as caught: - to_spec(override(model, **patch)) + to_spec(varied(model, **patch)) return str(caught.value) @@ -175,7 +175,7 @@ def test_an_unreferenced_nonlinear_entry_loads_and_is_reported(self): definition like any other — rather than degree-checking a declaration nothing consumes. """ - model = override(SMALL_MODEL, expressions={'lcoe': 'c / sum(p)'}) + model = varied(SMALL_MODEL, expressions={'lcoe': 'c / sum(p)'}) assert to_program(model).named_expressions['lcoe'].in_math is False, ( 'the unread nonlinear body loads rather than being refused, and nothing in the math reads it' ) @@ -206,7 +206,7 @@ def _kwarg_model(expression: str, dims: list[str] | None = None) -> dict[str, An class TestDual: """`dual(c)`: a primitive legal only in an entry the math never reads, its argument a constraint name resolved against constraints alone.""" - BASE = override(SMALL_MODEL, **{'constraints.lim': {'dims': ['g'], 'expression': 'p <= c'}}) + BASE = varied(SMALL_MODEL, **{'constraints.lim': {'dims': ['g'], 'expression': 'p <= c'}}) @pytest.mark.parametrize( ('patch', 'fragments'), @@ -266,13 +266,13 @@ class TestDual: ) def test_a_dual_out_of_place_is_refused(self, patch, fragments): with pytest.raises(LanguageError) as exc: - to_spec(override(self.BASE, **patch)) + to_spec(varied(self.BASE, **patch)) for fragment in fragments: assert fragment in str(exc.value) def test_a_dual_loads_in_an_expressions_entry(self): """The one place it is legal: an ``expressions:`` entry naming a declared constraint, which nothing in the math reads.""" - assert to_spec(override(self.BASE, expressions={'price': 'dual(lim)'})).expressions['price'] + assert to_spec(varied(self.BASE, expressions={'price': 'dual(lim)'})).expressions['price'] class TestDimensionKwargs: @@ -1427,15 +1427,13 @@ def test_an_expression_too_deep_to_walk_fails_as_a_language_error(patch, nests): nothing naming the file, the declaration, or what to write instead. """ with pytest.raises(LanguageError, match='past the 100 levels'): - to_spec(override(DISPATCH_MODEL, **patch)) + to_spec(varied(DISPATCH_MODEL, **patch)) def test_a_name_may_open_with_an_underscore(): """`expressions.md` said a name opens with a letter while the schema and the grammar both admitted `_`, so the page refused what the language accepts.""" schema = to_spec( - override( - DISPATCH_MODEL, **{'parameters._reserve': {'dims': ['generator']}, 'variables.p.where': '_reserve > 0'} - ) + varied(DISPATCH_MODEL, **{'parameters._reserve': {'dims': ['generator']}, 'variables.p.where': '_reserve > 0'}) ) assert '_reserve' in schema.parameters, 'a leading underscore is a name, as NAME and the schema both say' @@ -1466,7 +1464,7 @@ def record(*args, **kwargs): monkeypatch.setattr(module, door.__name__, recorded(door)) spec = to_spec( - override( + varied( DISPATCH_MODEL, **{ 'variables.p.where': 'p_max > 0', diff --git a/tests/typesetting/test_cases.py b/tests/typesetting/test_cases.py index 70da82ee..88039218 100644 --- a/tests/typesetting/test_cases.py +++ b/tests/typesetting/test_cases.py @@ -15,7 +15,7 @@ from math_spec.piecewise import expand_piecewise from math_spec.typesetting.symbols import chosen_expressions from tests.fixtures import DISPATCH_MODEL as DISPATCH -from tests.fixtures import override +from tests.fixtures import varied from tests.typesetting.fixtures import EVERY_FORMAT if TYPE_CHECKING: @@ -31,7 +31,7 @@ } #: The dispatch model, with a quantity defined by region and a constraint using it. -CASED = override( +CASED = varied( DISPATCH, **{ 'expressions.headroom': BY_REGION, @@ -41,7 +41,7 @@ #: One cased expression reached only through another's case. `opening_cost` has #: no variable of its own — its route to one runs through `headroom`. -_NESTED = override( +_NESTED = varied( CASED, **{ 'expressions.headroom.cases.opening.expression': 'p', @@ -83,7 +83,7 @@ def test_the_last_arm_prints_as_the_fallback_rather_than_a_condition(name: Forma @EVERY_FORMAT def test_a_declared_definition_prints_whether_or_not_a_row_names_it(name: FormatName, fmt: Format): """The rule a variable's domain follows: the file declared it, so it prints.""" - unused = override(CASED, **{'constraints.spare.expression': 'p <= p_max'}) + unused = varied(CASED, **{'constraints.spare.expression': 'p <= p_max'}) rendered = typeset(unused, name, legend=False) assert rendered.count(fmt.subscript(fmt.upright('headroom'), ['t', 'g'])) == 1, 'the definition, and no use' assert 'Definitions' in rendered @@ -111,7 +111,7 @@ def test_a_cased_expression_is_chosen_when_a_value_reaching_it_is( returns, and one case holding a variable is enough. The `otherwise:` is a value of the quantity like any case's, so a walk reading only the cases prints a solved quantity upright.""" - rendered = typeset(override(CASED, **patch), name, legend=False) + rendered = typeset(varied(CASED, **patch), name, legend=False) italic, upright = (fmt.subscript(face('headroom'), ['t', 'g']) for face in (fmt.italic, fmt.upright)) assert (italic in rendered) is chosen, 'the quantity is chosen exactly when a value reaching it holds a variable' assert (upright in rendered) is not chosen, 'and given otherwise, however its regions are chosen' @@ -145,7 +145,7 @@ def test_the_table_may_rename_a_named_expression_cased_or_plain(): tex = to_latex(CASED, symbols={'notation': 'latex', 'names': {'headroom': r'\bar h'}}, legend=False) assert r'\bar h_{t,g}' in tex - plain = override(DISPATCH, **{'expressions.supply': 'sum(p, over=generator)'}) + plain = varied(DISPATCH, **{'expressions.supply': 'sum(p, over=generator)'}) tex = to_latex(plain, symbols={'notation': 'latex', 'names': {'supply': 's'}}, legend=False) assert 's_{t} & =' in tex, 'the definition prints under the spelling the table gave' @@ -159,7 +159,7 @@ def test_the_definitions_print_in_declaration_order(): carrying two of them would churn on every regeneration. """ declared = ['alpha', 'bravo', 'charlie', 'delta', 'echo', 'foxtrot'] - tex = to_latex(override(CASED, **{f'expressions.{n}': BY_REGION for n in declared}), legend=False) + tex = to_latex(varied(CASED, **{f'expressions.{n}': BY_REGION for n in declared}), legend=False) section = tex[tex.index('Definitions') : tex.index('Variable domains')] labels = re.findall(r'^\\text\{(\w+)\} &&', section, flags=re.MULTILINE) assert labels == ['headroom', *declared], "declaration order, the file's own" diff --git a/tests/typesetting/test_declaration.py b/tests/typesetting/test_declaration.py index 2a537f9e..af534b1d 100644 --- a/tests/typesetting/test_declaration.py +++ b/tests/typesetting/test_declaration.py @@ -12,7 +12,7 @@ from math_spec import LanguageError, SchemaError, typeset_declaration from tests.fixtures import DISPATCH_MODEL as DISPATCH -from tests.fixtures import override +from tests.fixtures import varied from tests.typesetting.fixtures import EVERY_FORMAT from tests.typesetting.test_cases import CASED @@ -22,7 +22,7 @@ #: A variable-carrying reduction, a scalar reduction, a data-only body, and a #: constraint reading the first — the shapes a line has to read. -PLAIN = override( +PLAIN = varied( DISPATCH, **{ 'expressions.spend': 'sum(p * cost, over=generator)', @@ -114,7 +114,7 @@ def test_a_symbol_table_renames_an_expression_either_way(): def test_a_body_naming_another_expression_inlines_it_on_its_own_and_names_it_in_the_document(): """On its own, `double_spend` is complete only with `spend` substituted; in the document both are defined, each once, so a use prints the symbol.""" - model = override(PLAIN, **{'expressions.double_spend': 'spend * 2'}) + model = varied(PLAIN, **{'expressions.double_spend': 'spend * 2'}) assert typeset_declaration(model, 'double_spend', 'latex') == ( r'\mathit{double\_spend}_{t} = \left( \sum_{g \in \mathcal{G}} p_{t,g} \cdot \mathrm{cost}_{g} \right) ' r'\cdot 2 \qquad \forall\, t \in \mathcal{T}' @@ -125,7 +125,7 @@ def test_a_body_naming_another_expression_inlines_it_on_its_own_and_names_it_in_ #: A column and a row family this file reads, each named by something that prints. -GIVEN = override( +GIVEN = varied( PLAIN, **{ 'given.variables.flow': {'dims': ['snapshot']}, @@ -156,7 +156,7 @@ def test_a_name_that_prints_no_line_of_its_own_is_refused(model: dict[str, Any], def test_a_name_shared_by_a_constraint_and_a_variable_is_refused_rather_than_guessed(): """Constraints sit outside the flat namespace, so the model admits the pair; one line prints one of them.""" - model = override(PLAIN, **{'constraints.p': {'dims': ['snapshot', 'generator'], 'expression': 'p <= 1'}}) + model = varied(PLAIN, **{'constraints.p': {'dims': ['snapshot', 'generator'], 'expression': 'p <= 1'}}) with pytest.raises(SchemaError, match="'p' is both a constraint and a variable"): typeset_declaration(model, 'p', 'latex') @@ -167,6 +167,6 @@ def test_a_format_nobody_spells_is_refused(): def test_an_invalid_model_is_refused_before_anything_renders(): - broken = override(PLAIN, **{'expressions.spend': 'p * nonexistent'}) + broken = varied(PLAIN, **{'expressions.spend': 'p * nonexistent'}) with pytest.raises(LanguageError): typeset_declaration(broken, 'spend', 'latex') diff --git a/tests/typesetting/test_formats.py b/tests/typesetting/test_formats.py index 12498e5f..da438d3d 100644 --- a/tests/typesetting/test_formats.py +++ b/tests/typesetting/test_formats.py @@ -14,7 +14,7 @@ from math_spec import to_spec, typeset_declaration from math_spec.typesetting import FORMATS, to_latex, to_markdown, to_typst, typeset from math_spec.typesetting.format import OPERATOR_NAMES -from tests.fixtures import DISPATCH_MODEL, override +from tests.fixtures import DISPATCH_MODEL, varied from tests.typesetting import golden from tests.typesetting.fixtures import EVERY_FORMAT, TYPST_SYMBOLS @@ -116,7 +116,7 @@ def test_every_typst_operator_compiles(typst, tmp_path: Path): def test_the_model_description_opens_the_document(name: FormatName, fmt: Format, options: dict): """What the file says it is, printed before anything it declares — and printed with `legend=False` too, since it is not a symbol table.""" - described = override(DISPATCH_MODEL, description='least-cost dispatch of a generator fleet') + described = varied(DISPATCH_MODEL, description='least-cost dispatch of a generator fleet') out = typeset(described, name, **options) assert 'least-cost dispatch of a generator fleet' in out assert out.index('least-cost dispatch') < out.index(fmt.operators['minimize']), 'it opens the document' @@ -171,7 +171,7 @@ def test_a_description_sets_as_text_rather_than_as_markup(notation: str, positio one. """ where = 'description' if position == 'file' else 'parameters.load.description' - out = typeset(override(DISPATCH_MODEL, **{where: SPECIALS}), notation) + out = typeset(varied(DISPATCH_MODEL, **{where: SPECIALS}), notation) for expected in ESCAPED[notation]: assert expected in out, 'each special is escaped, and a character the notation reads as text is left alone' assert SPECIALS not in out, 'the raw prose reached the document unescaped' @@ -197,7 +197,7 @@ def test_a_backticked_name_in_a_description_sets_in_monospace(notation: str): span is part of the language's reading of prose rather than a Markdown habit that two formats printed as characters (#401). """ - out = typeset(override(DISPATCH_MODEL, **{'parameters.load.description': SPANNED}), notation) + out = typeset(varied(DISPATCH_MODEL, **{'parameters.load.description': SPANNED}), notation) assert SPANNED_AS[notation] in out, ( 'the span is monospace, its underscore escaped where the format needs it, and the lone backtick a character' ) @@ -207,7 +207,7 @@ def test_a_backticked_name_in_a_description_sets_in_monospace(notation: str): 'model', [ pytest.param(golden.MODEL, id='the-golden-model'), - pytest.param(override(DISPATCH_MODEL, description=SPECIALS), id='every-special'), + pytest.param(varied(DISPATCH_MODEL, description=SPECIALS), id='every-special'), ], ) def test_a_description_of_every_special_compiles(typst, tmp_path: Path, model): @@ -223,7 +223,7 @@ def test_a_description_of_every_special_compiles(typst, tmp_path: Path, model): def test_typst_prose_escapes_what_typst_reads_as_markup(typst, tmp_path: Path): - described = override(DISPATCH_MODEL, description='- a list? a // comment [a link] and = a heading') + described = varied(DISPATCH_MODEL, description='- a list? a // comment [a link] and = a heading') typ = to_typst(described, standalone=True) assert r'\- a list? a \/\/ comment \[a link\] and = a heading' in typ, ( 'a leading list marker, a comment and a link are escaped, and an inline `=` is no heading' @@ -234,7 +234,7 @@ def test_typst_prose_escapes_what_typst_reads_as_markup(typst, tmp_path: Path): def test_markdown_glossary_cells_survive_a_pipe_and_a_newline(): - described = override(DISPATCH_MODEL, **{'parameters.load.description': 'a | b\nc'}) + described = varied(DISPATCH_MODEL, **{'parameters.load.description': 'a | b\nc'}) md = to_markdown(described) assert r'| `load` over $`\mathcal{T}`$ — a \| b c |' in md, ( 'the pipe is escaped and the newline folded, so the cell stays one cell' diff --git a/tests/typesetting/test_symbols.py b/tests/typesetting/test_symbols.py index cabd21d4..5a0e6739 100644 --- a/tests/typesetting/test_symbols.py +++ b/tests/typesetting/test_symbols.py @@ -12,7 +12,7 @@ from math_spec.errors import SchemaError from math_spec.typesetting import SymbolTable, to_latex, to_markdown, to_typst, typeset -from tests.fixtures import DISPATCH_MODEL, override +from tests.fixtures import DISPATCH_MODEL, varied from tests.typesetting.fixtures import EVERY_FORMAT, TYPST_SYMBOLS if TYPE_CHECKING: @@ -20,7 +20,7 @@ from math_spec.typesetting.format import Format -WITH_MARGINAL_COST = override( +WITH_MARGINAL_COST = varied( DISPATCH_MODEL, **{'parameters.marginal_cost': {'dims': ['generator']}, 'objective.expression': 'sum(p * marginal_cost)'}, ) @@ -55,7 +55,7 @@ def test_the_table_prints_verbatim_and_the_rest_is_still_derived(render, symbols assert fragment in out -DESCRIBED = override( +DESCRIBED = varied( DISPATCH_MODEL, **{ 'dimensions.generator.description': 'dispatchable units', diff --git a/tests/typesetting/test_walk.py b/tests/typesetting/test_walk.py index 9f58968b..fc41bc44 100644 --- a/tests/typesetting/test_walk.py +++ b/tests/typesetting/test_walk.py @@ -17,7 +17,7 @@ from math_spec.typesetting.format import OPERATOR_NAMES from math_spec.typesetting.symbols import Symbols, _derive_name_symbol, chosen_expressions from math_spec.validation import to_spec -from tests.fixtures import DISPATCH_MODEL, OPERATOR_PROBES, override +from tests.fixtures import DISPATCH_MODEL, OPERATOR_PROBES, varied from tests.typesetting import golden from tests.typesetting.fixtures import EVERY_FORMAT, LATEX @@ -56,7 +56,7 @@ def test_a_dimension_index_never_steals_a_letter_a_variable_owns(name: FormatNam @EVERY_FORMAT def test_a_where_lands_on_the_quantifier_not_in_the_equation(name: FormatName, fmt: Format): """A mask is row absence, so it belongs to the ∀ that names the rows.""" - model = override(DISPATCH_MODEL, **{'variables.p.where': 'p_max > 0'}) + model = varied(DISPATCH_MODEL, **{'variables.p.where': 'p_max > 0'}) text = typeset(model, name, legend=False) forall, such_that = fmt.operators['forall'], fmt.operators['such_that'] masked = [line for line in text.splitlines() if such_that in line] @@ -232,7 +232,7 @@ def test_translations_that_disagree_at_the_edge_do_not_merge(name: FormatName, f @EVERY_FORMAT def test_a_negation_under_a_plus_is_the_subtraction_it_means(name: FormatName, fmt: Format): """`a + -b` is a spelling nobody uses, and the walk was printing it.""" - model = override(DISPATCH_MODEL, **{'objective.expression': 'sum(p) + -sum(p)'}) + model = varied(DISPATCH_MODEL, **{'objective.expression': 'sum(p) + -sum(p)'}) text = typeset(model, name) assert f'{fmt.operators["plus"]} {fmt.operators["minus"]}' not in text, 'a plus over a negation is a subtraction' assert fmt.operators['minus'] in text, 'the subtraction it folded into should still print' @@ -246,10 +246,10 @@ def test_a_mask_that_is_only_true_prints_no_condition(name: FormatName, fmt: For Nested it printed — `\\top \\wedge x` — while the program lowered the same mask to `x`: two readers of one file disagreeing about what it says. """ - always = override(DISPATCH_MODEL, **{'constraints.balance.where': 'True'}) + always = varied(DISPATCH_MODEL, **{'constraints.balance.where': 'True'}) assert typeset(always, name) == typeset(DISPATCH_MODEL, name), 'a mask every row passes is no mask at all' - nested = override(DISPATCH_MODEL, **{'constraints.balance.where': 'True AND load > 0'}) - plain = override(DISPATCH_MODEL, **{'constraints.balance.where': 'load > 0'}) + nested = varied(DISPATCH_MODEL, **{'constraints.balance.where': 'True AND load > 0'}) + plain = varied(DISPATCH_MODEL, **{'constraints.balance.where': 'load > 0'}) assert typeset(nested, name) == typeset(plain, name), 'a literal under a connective is folded before it prints' @@ -363,7 +363,7 @@ def test_a_description_is_joined_to_its_name_by_a_dash_the_format_renders(name: dash in two of the three outputs and as three hyphens in the one whose whole promise is that it renders where it lands. """ - described = override(DISPATCH_MODEL, **{'parameters.cost.description': 'marginal cost'}) + described = varied(DISPATCH_MODEL, **{'parameters.cost.description': 'marginal cost'}) text = typeset(described, name) assert f'{fmt.dash} marginal cost' in text if fmt is FORMATS['markdown']: @@ -375,7 +375,7 @@ def test_a_named_expression_prints_once_as_a_definition_and_by_symbol_where_used """The file names the quantity, so the page does: a use prints the symbol and the body prints once under Definitions. A macro is sugar with no identity of its own, so it is expanded away either way.""" - model = override( + model = varied( DISPATCH_MODEL, **{'expressions.supply': 'sum(p, over=generator)', 'constraints.balance.expression': 'supply == load'}, ) @@ -387,7 +387,7 @@ def test_a_named_expression_prints_once_as_a_definition_and_by_symbol_where_used @EVERY_FORMAT def test_inlining_substitutes_a_named_expression_where_it_is_used(name: FormatName, fmt: Format): """What prints then is the math a backend builds, not the name it was spelled with.""" - model = override( + model = varied( DISPATCH_MODEL, **{'expressions.supply': 'sum(p, over=generator)', 'constraints.balance.expression': 'supply == load'}, ) @@ -398,7 +398,7 @@ def test_inlining_substitutes_a_named_expression_where_it_is_used(name: FormatNa @EVERY_FORMAT def test_an_invalid_model_is_refused_before_anything_renders(name: FormatName, fmt: Format): - broken = override(DISPATCH_MODEL, **{'objective.expression': 'p * nonexistent'}) + broken = varied(DISPATCH_MODEL, **{'objective.expression': 'p * nonexistent'}) with pytest.raises(LanguageError): typeset(broken, name) @@ -408,7 +408,7 @@ def test_inlining_keeps_the_definition_of_an_entry_the_math_never_reads(name: Fo """Substitution has nowhere to put it: nothing in the objective or a constraint names it, so dropping its definition would drop the quantity from the page entirely.""" - model = override( + model = varied( DISPATCH_MODEL, **{ 'expressions.supply': 'sum(p, over=generator)', @@ -426,7 +426,7 @@ def test_a_dual_prints_the_constraint_symbol_not_a_same_named_variable(name: For """`dual(c)` subscripts λ from a map of its own, so a variable sharing the constraint's name — a legal collision, constraints sit outside the flat namespace (#74) — cannot lend the dual its italic letter.""" - model = override( + model = varied( DISPATCH_MODEL, **{ 'variables.balance': {'dims': ['snapshot'], 'bounds': {'lower': 0}}, @@ -447,7 +447,7 @@ def test_an_entry_reading_a_dual_prints_italic(name: FormatName, fmt: Format): """Upright is what the model is given, and a shadow price is not: no data hands it over, the solve settles it — the same reason a variable is italic, though a dual carries no variable for `carries_variable` to find.""" - model = override(DISPATCH_MODEL, **{'expressions.mp': 'dual(balance)'}) + model = varied(DISPATCH_MODEL, **{'expressions.mp': 'dual(balance)'}) assert fmt.subscript(fmt.italic('mp'), ['t']) in typeset(model, name, legend=False), ( 'the entry is read off the solution, so its own symbol is italic' ) @@ -496,7 +496,7 @@ def test_a_given_quantity_is_upright(name: str, expected: str): def test_a_name_that_is_a_greek_letter_prints_as_the_letter(name: FormatName, fmt: Format): """A variable called `theta` set as the italic word *theta* is the one derived symbol no paper would accept.""" - model = override(DISPATCH_MODEL, **{'variables.theta': {'dims': ['snapshot']}}) + model = varied(DISPATCH_MODEL, **{'variables.theta': {'dims': ['snapshot']}}) assert fmt.greek('theta') in typeset(model, name) @@ -552,7 +552,7 @@ def test_a_dimension_is_not_a_head_a_qualifier_hangs_off(name: FormatName, fmt: whether some unrelated dimension happened to share its prefix: declare a dimension named `tech` and `tech_cap` silently re-rendered. """ - model = override( + model = varied( DISPATCH_MODEL, **{'dimensions.zone': {'dtype': 'str'}, 'parameters.zone_cap': {'dims': ['zone']}}, ) @@ -608,16 +608,14 @@ def test_the_objective_shows_the_summations_the_file_wrote(name: FormatName, fmt @EVERY_FORMAT def test_two_sums_of_the_same_dims_stay_two_summations(name: FormatName, fmt: Format): """The file's structure survives to the page, even where it repeats itself.""" - text = typeset(override(MIXED, **{'objective.expression': 'sum(p * cost) + sum(p * cost)'}), name, legend=False) + text = typeset(varied(MIXED, **{'objective.expression': 'sum(p * cost) + sum(p * cost)'}), name, legend=False) assert summations(text, fmt) == 2, 'two written sums are two summations' @EVERY_FORMAT def test_a_subtracted_summation_keeps_the_sign_outside_it(name: FormatName, fmt: Format): """The sign is applied to the whole reduction, and the bracket says so.""" - text = typeset( - override(MIXED, **{'objective.expression': 'sum(p * cost) - sum(p_nom * capex)'}), name, legend=False - ) + text = typeset(varied(MIXED, **{'objective.expression': 'sum(p * cost) - sum(p_nom * capex)'}), name, legend=False) opener = fmt.parenthesise('BODY').split('BODY')[0] + over_generators(fmt) assert f'{fmt.operators["minus"]} {opener}' in text @@ -663,7 +661,7 @@ def test_every_operator_probe_renders(path, name: FormatName, fmt: Format): def _grouped(dims: list[str], expression: str) -> str: """The constraint `c` over *dims*, as the one line of LaTeX it prints.""" - model = override(UNREAD, **{'constraints.c': {'dims': dims, 'expression': expression}}) + model = varied(UNREAD, **{'constraints.c': {'dims': dims, 'expression': expression}}) return next(line for line in to_latex(model, legend=False).splitlines() if line.startswith(r'\text{c}')) @@ -713,7 +711,7 @@ def test_a_value_column_the_walk_consumes_is_a_condition_like_a_produced_one(): def _row(expression: str, where: str | None = None, **patch: object) -> str: - model = override( + model = varied( BUSES, **{'constraints.k': {'dims': ['snapshot', 'generator'], 'expression': expression, 'where': where}}, **patch, From e65969fd650521984f0add2dc234fda909ba1416 Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 17:17:50 +0200 Subject: [PATCH 4/9] fix(language): a patch cannot retire or partly restate a dimension, and a null section is refused `override` read `dimensions: {snapshot: null}` as a removal and `dimensions: {snapshot: {}}` as a restatement, though the rule is that a patch adds a dimension or a relation or restates one word for word. Both are refused now, with a message that names the rewrite. A section set to null, such as `constraints: null`, laid as an empty mapping and changed nothing. It is refused too, and the message says that removal is written one declaration at a time. The page and the messages now say dimension and relation throughout, where they said axis in some places and dimension in others. --- docs/about/limits.md | 4 +- docs/howto/compose.md | 37 ++++++++++------ src/math_spec/composition.py | 77 +++++++++++++++++++++------------ tests/test_composition.py | 84 +++++++++++++++++++++++++++++------- 4 files changed, 145 insertions(+), 57 deletions(-) diff --git a/docs/about/limits.md b/docs/about/limits.md index d6df2788..cee468df 100644 --- a/docs/about/limits.md +++ b/docs/about/limits.md @@ -159,6 +159,6 @@ declare a `p`, are both things a library does before it hands over a `dict`. A project that extends a model it does not own writes a patch, not a copy. `override` lays the patch over the base a field at a time, and refuses a patch -that lands on nothing, two patches that write one field, and an axis redeclared -under the math. The recipe is in +that lands on nothing, two patches that write one field, and a dimension or a +relation redeclared under the math. The recipe is in [compose a model from several files](../howto/compose.md). diff --git a/docs/howto/compose.md b/docs/howto/compose.md index b382983c..3794ad80 100644 --- a/docs/howto/compose.md +++ b/docs/howto/compose.md @@ -75,7 +75,8 @@ file. The marker is the declaration itself. Deeper down, `null` is a value the schema takes: `dispatch: { where: null }` gives that variable no mask, and - leaves the variable in place. + leaves the variable in place. Higher up, `constraints: null` is refused, + because a section is not a declaration and nulling it removes nothing. 4. **Nest the calls where one patch refines another.** The second call lays its patch on the first call's result, so the order is on the page. @@ -86,14 +87,15 @@ file. ## What a patch may say -| The entry | What happens | -| ------------------------------------ | ---------------------------------------------------------- | -| some fields of a declaration | those fields change, and the rest of the declaration stays | -| a whole declaration under a new name | it is added | -| `null` under a declaration's name | it is removed | -| a dimension or a relation | it is added, or restated exactly as the base declares it | +| The entry | What happens | +| ----------------------------------------------------------- | ----------------------------------------------------------------------------- | +| some fields of a declaration | those fields change, and the rest of the declaration stays | +| a whole declaration under a new name | it is added | +| `null` under a declaration's name | it is removed | +| `null` under a section's name | it is refused | +| a dimension or a relation | it is added, or restated word for word as the base declares it | | an entry under `given: variables:` or `given: constraints:` | it is edited, added or removed like any declaration, and the other kind stays | -| `version`, `description` | the patch's value replaces the base's | +| `version`, `description` | the patch's value replaces the base's | ## A partial entry on a missing name @@ -115,14 +117,25 @@ Two patches writing one field is refused, both named: patches 'pathway' and 'project': both write variables.dispatch.bounds.upper. Patches laid on one base are disjoint, so nothing decides which of two writes wins. Write the change in one patch, or lay one patch on the result of the other: override(override(base, {'pathway': …}), {'project': …}). ``` -## An axis redeclared +## A dimension redeclared A patch may add a dimension or a relation, and may restate one the base -declares. Changing one under the expressions already written over it is -refused: +declares. The restatement is word for word: half a declaration is a second +reading of the same name. Changing one under the expressions already written +over it is refused, and so is removing one: ```text -patch 'relabelled' declares the dimension 'snapshot' as {'dtype': 'str'}, where its base declares {'dtype': 'int'}. A patch adjusts the math, not the axes the math is already written over: restate the declaration exactly, leave it out, or give the patch an axis of its own under a name of its own. +patch 'relabelled' declares the dimension 'snapshot' as {'dtype': 'str'}, where its base declares {'dtype': 'int'}. A patch adjusts the math, not the coordinate space the math is already written over: restate the declaration word for word, leave it out, or give the patch a dimension of its own under a name of its own. +``` + +## A section set to `null` + +A `null` removes the declaration it names. A section holds declarations rather +than being one, so nulling a section is refused rather than read as emptying +it: + +```text +patch 'project' sets 'constraints' to null, which removes nothing: the removal marker names one declaration, and a section is not one. Remove the declarations one at a time, each under its own name, or leave the section out of the patch. ``` ## A stale removal diff --git a/src/math_spec/composition.py b/src/math_spec/composition.py index db34d5c7..467d4dd6 100644 --- a/src/math_spec/composition.py +++ b/src/math_spec/composition.py @@ -24,12 +24,14 @@ * **Sibling patches are disjoint.** Two patches writing one field is refused, both named, so the order they are given in never decides a model. Layering is written out as ``override(override(base, …), …)``. -* **A patch adjusts the math, not the axes.** A ``dimensions`` or ``relations`` - entry may be added or restated exactly, never changed. +* **A patch adjusts the math, not the coordinate space.** A ``dimensions`` or + ``relations`` entry may be added or restated word for word, never changed and + never removed. * **A declaration set to** ``null`` **is removed**, and a removal of what the base does not declare is refused. The marker is positional: ``constraints: {ramp: null}`` removes the constraint, where ``variables: {p: {where: null}}`` - sets that variable's mask to none, which is a value the schema takes. + sets that variable's mask to none, which is a value the schema takes. A whole + section set to ``null`` is refused, because it removes nothing. * **``given:`` is laid over one kind at a time**, by the same rules as any owned section. """ @@ -50,8 +52,8 @@ from pathlib import Path #: The declarations that are the coordinate space rather than the math. A patch -#: may add one, and may restate one its base already declares; it may not say -#: something else about it. +#: may add one, and may restate one its base already declares word for word; it +#: may not say something else about it, and it may not remove it. SHARED_SECTIONS = ('dimensions', 'relations') #: The declarations a patch edits, creates or removes. @@ -93,8 +95,8 @@ def override( Raises: LanguageError: A patch edits or removes a declaration its base does not declare; a patch creates one that is not whole; a patch redeclares - a dimension or a relation as something else; or two patches write - one field. + or removes a dimension or a relation; a patch sets a whole section + to ``null``; or two patches write one field. FileNotFoundError: A ``str`` with no newline that names no file. """ read = {name: _declarations(patch) for name, patch in patches.items()} @@ -225,11 +227,12 @@ def _lay_over(base: dict[str, Any], patch: dict[str, Any], name: str) -> dict[st laid = dict(base) for key, value in patch.items(): if key == 'given': - laid[key] = _given(laid.get(key) or {}, value or {}, name) + laid[key] = _given(laid.get(key) or {}, _section(value, key, name), name) elif key in SHARED_SECTIONS: - laid[key] = _shared(laid.get(key) or {}, value or {}, key, name) + laid[key] = _shared(laid.get(key) or {}, _section(value, key, name), key, name) elif key in OWNED_SECTIONS: - laid[key] = _owned(laid.get(key) or {}, value or {}, _singular(key), _entry_class(Spec, key), name) + block = _section(value, key, name) + laid[key] = _owned(laid.get(key) or {}, block, _singular(key), _entry_class(Spec, key), name) elif key == 'objective': laid = _objective(laid, value, name) else: @@ -237,6 +240,22 @@ def _lay_over(base: dict[str, Any], patch: dict[str, Any], name: str) -> dict[st return laid +def _section(value: Any, where: str, name: str) -> dict[str, Any]: + """The block a patch writes under one section, a ``null`` section being refused rather than read as empty. + + A section is not a declaration, so the removal marker does not reach it. An + empty mapping laid over a base says nothing either, and this is the spelling + a writer reaches for when they mean to empty the section. + """ + if value is None: + raise LanguageError( + f"patch '{name}' sets '{where}' to null, which removes nothing: the removal marker names one " + f'declaration, and a section is not one. Remove the declarations one at a time, each under its ' + f'own name, or leave the section out of the patch.' + ) + return cast('dict[str, Any]', value) + + def _given(declared: dict[str, Any], patch: dict[str, Any], name: str) -> dict[str, Any]: """The ``given:`` block, one kind laid over at a time, so naming the columns keeps the row families. @@ -247,39 +266,41 @@ def _given(declared: dict[str, Any], patch: dict[str, Any], name: str) -> dict[s for kind, block in patch.items(): if kind in GIVEN_KINDS: cls = _entry_class(GivenBlock, kind) - out[kind] = _owned(out.get(kind) or {}, block or {}, GIVEN_KINDS[kind], cls, name) + entries = _section(block, f'given: {kind}:', name) + out[kind] = _owned(out.get(kind) or {}, entries, GIVEN_KINDS[kind], cls, name) else: out[kind] = block return out def _shared(declared: dict[str, Any], patch: dict[str, Any], section: str, name: str) -> dict[str, Any]: - """One ``dimensions`` or ``relations`` block: a patch adds an axis or restates one, never changes it.""" + """One ``dimensions`` or ``relations`` block: a patch adds one or restates one, never changes or drops it. + + The restatement is compared for equality rather than field by field: a + patch that names half a declaration is as much a second reading of the + coordinate space as one that names another value. + """ out = dict(declared) + singular = _singular(section) for key, block in patch.items(): if block is None: - _removed(out, key, _singular(section), name) - elif key not in out: + raise LanguageError( + f"patch '{name}' removes the {singular} '{key}'. The coordinate space is what the math is " + f'written over, and a patch adjusts the math rather than the space: leave the {singular} out ' + f'of the patch, and remove the declarations written over it one at a time.' + ) + if key not in out: out[key] = block - elif not _agrees(out[key], block): + elif out[key] != block: raise LanguageError( - f"patch '{name}' declares the {_singular(section)} '{key}' as {block!r}, where its base " - f'declares {out[key]!r}. A patch adjusts the math, not the axes the math is already ' - f'written over: restate the declaration exactly, leave it out, or give the patch an ' - f'axis of its own under a name of its own.' + f"patch '{name}' declares the {singular} '{key}' as {block!r}, where its base " + f'declares {out[key]!r}. A patch adjusts the math, not the coordinate space the math is ' + f'already written over: restate the declaration word for word, leave it out, or give the ' + f'patch {_a(singular)} of its own under a name of its own.' ) return out -def _agrees(under: Any, over: Any) -> bool: - """Whether *over* says only what *under* already says, a field the patch omits being no claim.""" - if isinstance(over, dict): - return isinstance(under, dict) and all( - key in under and _agrees(under[key], value) for key, value in over.items() - ) - return bool(under == over) - - def _owned( declared: dict[str, Any], patch: dict[str, Any], label: str, cls: type[BaseModel], name: str ) -> dict[str, Any]: diff --git a/tests/test_composition.py b/tests/test_composition.py index e1e840e7..c872fbf8 100644 --- a/tests/test_composition.py +++ b/tests/test_composition.py @@ -6,9 +6,10 @@ A name the patch declares is the point, and what is pinned for it is the opposite: every collision the caller did not ask for is an error naming both -sides. A patch that lands on nothing, two patches writing one field, and an -axis redeclared under the expressions written over it are the three, and each -one is a model that would otherwise load and mean something nobody wrote. +sides. A patch that lands on nothing, two patches writing one field, a +dimension redeclared or removed under the expressions written over it, and a +whole section set to null are each a model that would otherwise load and mean +something nobody wrote. """ from __future__ import annotations @@ -27,6 +28,15 @@ 'constraints': {'co2_cap': {'dims': [], 'expression': 'sum(p * co2) <= 100'}}, } +#: A base that reads a solved model: one given column, one given constraint and +#: an expression over the constraint, so a patch to `given:` is checked by +#: `to_spec` rather than only laid. +GIVEN_BASE = { + 'dimensions': {'g': {'dtype': 'str'}}, + 'given': {'variables': {'p': {'dims': ['g']}}, 'constraints': {'cap': {'dims': ['g']}}}, + 'expressions': {'price': {'expression': 'dual(cap)'}}, +} + #: `DISPATCH_MODEL` with no objective, for the patches that ask about one. FEASIBILITY = {k: v for k, v in DISPATCH_MODEL.items() if k != 'objective'} @@ -177,20 +187,56 @@ def test_layering_is_written_out_as_nesting(): assert twice['variables']['p']['where'] == 'cost > 0' -def test_a_patch_adds_an_axis_and_may_restate_one_it_shares(): +def test_a_patch_adds_a_dimension_and_may_restate_one_it_shares(): laid = override( DISPATCH_MODEL, {'periods': {'dimensions': {'snapshot': {'dtype': 'int'}, 'investment_period': {'dtype': 'int'}}}}, ) assert sorted(laid['dimensions']) == ['generator', 'investment_period', 'snapshot'], ( - 'the axis the patch adds joins the two the base declares, and the restated one is not doubled' + 'the dimension the patch adds joins the two the base declares, and the restated one is not doubled' ) -def test_a_patch_that_redeclares_an_axis_is_refused(): - """An axis changed under the expressions already written over it is a different model, silently.""" - with pytest.raises(LanguageError, match=r'adjusts the math, not the axes'): - override(DISPATCH_MODEL, {'relabelled': {'dimensions': {'snapshot': {'dtype': 'str'}}}}) +@pytest.mark.parametrize( + ('patch', 'says'), + [ + pytest.param( + {'dimensions': {'snapshot': {'dtype': 'str'}}}, + 'adjusts the math, not the coordinate space', + id='declared-as-something-else', + ), + pytest.param( + {'dimensions': {'snapshot': {}}}, + 'restate the declaration word for word', + id='restated-in-part', + ), + pytest.param( + {'dimensions': {'snapshot': None}}, + 'remove the declarations written over it one at a time', + id='removed', + ), + ], +) +def test_a_patch_that_rewrites_a_dimension_is_refused(patch, says): + """A dimension changed under the expressions already written over it is a different model, silently.""" + with pytest.raises(LanguageError) as raised: + override(DISPATCH_MODEL, {'relabelled': patch}) + assert says in str(raised.value), 'the refusal names the rewrite rather than only what is wrong' + + +@pytest.mark.parametrize( + 'patch', + [ + pytest.param({'constraints': None}, id='an-owned-section'), + pytest.param({'dimensions': None}, id='a-shared-section'), + pytest.param({'given': {'variables': None}}, id='one-kind-of-given'), + ], +) +def test_a_whole_section_set_to_null_is_refused(patch): + """Nulling a section reads as emptying it, and laying it silently changed nothing at all.""" + with pytest.raises(LanguageError, match=r'removes nothing') as raised: + override(DISPATCH_MODEL, {'blank': patch}) + assert 'one at a time' in str(raised.value), 'the refusal names the rewrite, which is one null per declaration' def test_the_objective_is_laid_over_field_by_field(): @@ -228,12 +274,7 @@ def test_an_objective_a_base_does_not_declare_is_refused(patch, says): def test_a_patch_over_one_kind_of_given_leaves_the_other_alone(): """`given:` is laid over a kind at a time, so patching the columns cannot drop the row families.""" - base = { - 'dimensions': {'g': {'dtype': 'str'}}, - 'given': {'variables': {'p': {'dims': ['g']}}, 'constraints': {'cap': {'dims': ['g']}}}, - 'expressions': {'price': {'expression': 'dual(cap)'}}, - } - laid = override(base, {'wider': {'given': {'variables': {'p': {'domain': 'binary'}}}}}) + laid = override(GIVEN_BASE, {'wider': {'given': {'variables': {'p': {'domain': 'binary'}}}}}) assert laid['given']['variables']['p'] == {'dims': ['g'], 'domain': 'binary'}, ( 'the given column is edited field by field like any declaration' ) @@ -241,6 +282,19 @@ def test_a_patch_over_one_kind_of_given_leaves_the_other_alone(): assert to_spec(laid).given.variables['p'].domain == 'binary' +@pytest.mark.parametrize( + ('base', 'reads'), + [ + pytest.param(GIVEN_BASE, ['p', 'q'], id='a-base-that-already-reads'), + pytest.param({'dimensions': {'g': {'dtype': 'str'}}}, ['q'], id='a-base-that-reads-nothing-yet'), + ], +) +def test_a_whole_given_entry_is_created_and_the_model_loads(base, reads): + """A patch adds a column to read, whether or not the base opened the block.""" + laid = override(base, {'solved': {'given': {'variables': {'q': {'dims': ['g']}}}}}) + assert sorted(to_spec(laid).given.variables) == reads, 'the created column joins whatever the base read' + + def test_a_patch_is_a_path_as_readily_as_a_mapping(tmp_path): """Whatever every other verb takes, so a patch travels as a file rather than as a script.""" patch = tmp_path / 'carbon.yaml' From de802ddcf7a62d485bd85c5d4f4aef4ed056c4dd Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 17:26:01 +0200 Subject: [PATCH 5/9] feat(language): fragments compose into one model, each reading what a sibling introduces merge(fragments, description=None) composes peers before validation. A dimension or a relation every fragment may declare, and the ones that do say the same thing about it, prose excluded. Every other name is owned, and a second claim is refused naming both fragments. The objectives are summed, each term in parentheses, and the senses and the versions have to agree. A given declaration is folded into the declaration a sibling introduces, once the reader is checked to say the same or less, and two fragments that both only read a name have to read it the same way. merge is exported beside override, the given advice names merge() as the fragment's route, and compose.md, declarations.md and limits.md describe both verbs. --- docs/about/limits.md | 21 +-- docs/howto/compose.md | 136 ++++++++++++++++- docs/reference/language/declarations.md | 5 + src/math_spec/__init__.py | 3 +- src/math_spec/advice.py | 3 +- src/math_spec/composition.py | 189 +++++++++++++++++++++++- tests/test_composition.py | 156 ++++++++++++++++++- tests/test_given.py | 88 ++++++++++- tests/test_public_surface.py | 4 +- 9 files changed, 577 insertions(+), 28 deletions(-) diff --git a/docs/about/limits.md b/docs/about/limits.md index cee468df..3e049b06 100644 --- a/docs/about/limits.md +++ b/docs/about/limits.md @@ -151,14 +151,17 @@ column under [`given: variables:`](../reference/language/declarations.md#given). So it loads on its own, and prints as math on its own. -Merging happens before `to_spec`. Every function here takes a `dict` as well as -a path, so a model assembled in Python is checked exactly as a file is, and -`Spec.to_yaml()` writes the file a reviewer reads. A `dict` may hold only what a -file may hold. A built-in merge, and namespaces so that two fragments can each -declare a `p`, are both things a library does before it hands over a `dict`. - -A project that extends a model it does not own writes a patch, not a copy. -`override` lays the patch over the base a field at a time, and refuses a patch +Composition happens before `to_spec`, and two verbs do it. `merge` composes +fragments as peers: a name two of them declare is refused, and a given +declaration is folded into the fragment that introduces the name. `override` +lays a patch over a base a field at a time, which is what a project that +extends a model it does not own writes instead of a copy. It refuses a patch that lands on nothing, two patches that write one field, and a dimension or a -relation redeclared under the math. The recipe is in +relation redeclared under the math. The recipe for both is in [compose a model from several files](../howto/compose.md). + +Every function here takes a `dict` as well as a path, so a model assembled in +Python is checked exactly as a file is, and `Spec.to_yaml()` writes the file a +reviewer reads. A `dict` may hold only what a file may hold. Namespaces, so +that two fragments can each declare a `p`, are a library's business before it +hands over a `dict`. diff --git a/docs/howto/compose.md b/docs/howto/compose.md index 3794ad80..80c2850a 100644 --- a/docs/howto/compose.md +++ b/docs/howto/compose.md @@ -5,11 +5,139 @@ SPDX-License-Identifier: CC-BY-4.0 # Compose a model from several files -Build one model out of files that each say part of it. `override` lays -**patches** over a **base**: the model a framework ships, and the change a -project makes to it. It hands back one mapping, which +Build one model out of files that each say part of it. `merge` composes +**fragments**: the files of a component library, each owning part of the math. +`override` lays **patches** over a **base**: the model a framework ships, and +the change a project makes to it. Each hands back one mapping, which [`to_spec`](../reference/language/errors.md#what-to_spec-checks) loads like any -file. +file, and the two compose as `override(merge({…}), {…})`. + +## A library of components + +1. **Write the coupling surface as a model.** One flow per port, one balance + per bus. Nothing in it names a component type. + + ```yaml title="surface.yaml" + dimensions: + snapshot: { dtype: int } + bus: { dtype: str } + port: { dtype: str } + relations: + Port_bus: { key: port, values: bus } + variables: + Port_p: + dims: [snapshot, port] + description: what a port puts into its bus + constraints: + Bus_balance: + dims: [snapshot, bus] + expression: sum(Port_p, by=Port_bus, over=port, into=bus) == 0 + ``` + +2. **Write each component file against that surface.** It declares its own + dimension, its own math, and one relation into `port`. It names `Port_p` + under [`given`](../reference/language/declarations.md#given), because the + surface introduces that column and this file only reads it. + + ```yaml title="generator.yaml" + dimensions: + snapshot: { dtype: int } + port: { dtype: str } + generator: { dtype: str } + relations: + Generator_port: { key: generator, values: port } + given: + variables: + Port_p: { dims: [snapshot, port] } + parameters: + Generator_p_nom: { dims: [generator] } + Generator_marginal_cost: { dims: [generator] } + variables: + Generator_p: { dims: [snapshot, generator], bounds: { lower: 0, upper: Generator_p_nom } } + constraints: + Generator_injection: + dims: [snapshot, generator] + expression: at(Port_p, by=Generator_port, over=port, into=generator) == Generator_p + objective: + sense: minimize + expression: sum(Generator_p * Generator_marginal_cost) + ``` + + ```yaml title="load.yaml" + dimensions: + snapshot: { dtype: int } + port: { dtype: str } + load: { dtype: str } + relations: + Load_port: { key: load, values: port } + given: + variables: + Port_p: { dims: [snapshot, port] } + parameters: + Load_p_set: { dims: [snapshot, load] } + constraints: + Load_withdrawal: + dims: [snapshot, load] + expression: at(Port_p, by=Load_port, over=port, into=load) == -Load_p_set + ``` + + Each file loads on its own and prints as math on its own. + +3. **Merge the files you need.** Each fragment is given a name, and that name + is what a refusal calls it. The order the fragments are given in does not + change the model. + + ```python + import math_spec as ms + + model = ms.merge({'surface': 'surface.yaml', 'generator': 'generator.yaml', 'load': 'load.yaml'}) + spec = ms.to_spec(model) + ``` + + `merge` folds each given declaration into the declaration that introduces + it, so `spec` declares `Port_p` once and carries no `given:`. The objectives + of the fragments are summed, each term in parentheses. + +4. **Add a component type without touching the balance.** A component file + pins the flow at its own port rather than adding a term to the balance, so + `Bus_balance` is written once and stays as it is however many files are + merged. What grows is the data: which ports exist, and which bus each one + sits on. + +## What a fragment may share + +| The entry | What happens | +| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| a dimension or a relation | every fragment may declare it, and the ones that do say the same thing about it | +| a `description` on a shared dimension or relation | it is prose rather than a claim, and the first fragment's wording is carried | +| any other declaration | one fragment declares it, and a second is refused | +| an entry under `given: variables:` or `given: constraints:` | it is checked against the fragment that introduces the name, then folded into it | +| a given entry no fragment introduces | it stays under `given:` for a consumer to bind | +| `objective` | the terms are summed, each in parentheses, and the senses agree | +| `version` | every fragment says the same one, and a fragment that says none is version 0 | +| `description` at the top of a fragment | it is about the fragment and is not carried. Pass the composed model's as `description=` | + +## A name two fragments declare + +Fragments own their math, so a name two of them declare is refused, both +named. Here two files each say what a generator fleet is: + +```text +fragments 'gas' and 'coal' both declare the parameter 'Generator_p_nom'. Two of the same kind of thing are two rows of a dimension rather than two fragments: merge the fragment once, and let the data carry both. Different math under one spelling is a rename: call one of them something else. +``` + +## A column read one way and introduced another + +What a fragment states about a column it reads has to agree with the fragment +that introduces the column. The reader may say less, such as the frame with no +`domain`, and may not say something else: + +```text +fragment 'generator' reads the given variable 'Port_p' as {'dims': ['snapshot', 'generator']}, where 'surface' introduces it as {'dims': ['snapshot', 'port'], 'description': 'what a port puts into its bus'}. A given declaration says the same as the declaration it is folded into, or less: restate the frame as the introducer declares it, or leave the field out. +``` + +Two fragments that both only read a column have to read it the same way, and +a difference is refused as it is for a dimension. ## A base and its patches diff --git a/docs/reference/language/declarations.md b/docs/reference/language/declarations.md index d183c0aa..ad4e1762 100644 --- a/docs/reference/language/declarations.md +++ b/docs/reference/language/declarations.md @@ -128,6 +128,11 @@ An expression reads a given variable as it reads any other. A name declared under both `variables:` and `given: variables:` is refused. The typeset legend lists a given variable under _Given_, and prints no domain line for it. +[`merge`](../../howto/compose.md#a-library-of-components) folds a given +declaration into the declaration of another fragment that introduces the name, +so a composed library carries none of them. The folded declaration is the +introducer's, and what the reader states has to say the same or less. + Where nothing in this language introduces the column, the program carries the declaration for a consumer to bind ([what a program does not build](../reading.md#what-a-program-does-not-build)). diff --git a/src/math_spec/__init__.py b/src/math_spec/__init__.py index f281b21a..a20ec393 100644 --- a/src/math_spec/__init__.py +++ b/src/math_spec/__init__.py @@ -12,7 +12,7 @@ from math_spec import program from math_spec.advice import advice -from math_spec.composition import override +from math_spec.composition import merge, override from math_spec.errors import ( ADVICE_KINDS, Advice, @@ -75,6 +75,7 @@ 'call_shape_error', 'did_you_mean', 'edge_error', + 'merge', 'override', 'program', 'schema_error', diff --git a/src/math_spec/advice.py b/src/math_spec/advice.py index 1bc0035b..f538ead4 100644 --- a/src/math_spec/advice.py +++ b/src/math_spec/advice.py @@ -52,7 +52,8 @@ def _given(program: Program) -> list[Advice]: 'given', name, f"{kind} '{name}' is read here and built elsewhere: a consumer binds it to the model this " - f'one is layered onto, checks the frame, and refuses where it cannot bind it.', + f'one is layered onto, checks the frame, and refuses where it cannot bind it. A fragment is ' + f'composed instead: merge() folds it into the file that introduces it.', ) for kind, group in (('variable', program.given.variables), ('row family', program.given.constraints)) for name in group diff --git a/src/math_spec/composition.py b/src/math_spec/composition.py index 467d4dd6..e5e649cd 100644 --- a/src/math_spec/composition.py +++ b/src/math_spec/composition.py @@ -4,9 +4,30 @@ """Several files into one model, before any of them is validated. +Two verbs, and they answer different questions. :func:`merge` composes +**peers**: fragments that each own part of the math, where a name two of them +declare is a collision and the order they are given in means nothing. :func:`override` lays **patches** over a **base**: what a framework ships and a -project extends. A patch says only what it changes, because declarations are -laid over a field at a time:: +project extends, where a name the patch declares is the point. They compose as +``override(merge({...}), {...})``, which builds the model and then configures +the run. + +What :func:`merge` does with each section: + +* **A dimension or a relation every fragment may declare**, and the ones that + do have to say the same thing about it. Prose is not a claim, so two + descriptions of one dimension agree, and the first fragment's is carried. +* **Every other declaration is owned.** A name two fragments declare is refused, + both named. +* **The objectives are summed**, each term in parentheses, and the senses have + to agree. +* **A given declaration is folded** into the declaration that introduces the + name, once the reader is checked to say the same as the introducer or less. + Two fragments that both only read a name have to read it the same way. What + no fragment introduces stays under ``given:`` for a consumer to bind. + +A patch says only what it changes, because declarations are laid over a field +at a time:: constraints: ramp: {dims: [snapshot, generator, investment_period]} @@ -59,7 +80,9 @@ #: The declarations a patch edits, creates or removes. OWNED_SECTIONS = ('parameters', 'variables', 'constraints', 'expressions', 'macros', 'piecewise', 'sos') -#: What ``given:`` holds, by the key each kind sits under and what one entry of it is called. +#: What ``given:`` holds, by the key each kind sits under and what one entry of +#: it is called. The key is the introducing section's name too, which is what +#: lets :func:`merge` fold a given declaration into the one that introduces it. GIVEN_KINDS = {'variables': 'given variable', 'constraints': 'given constraint'} #: Every section keyed by declaration name. ``objective`` is one declaration @@ -74,6 +97,166 @@ } +def merge( + fragments: Mapping[str, str | Path | dict[str, Any] | Spec], description: str | None = None +) -> dict[str, Any]: + """*fragments* composed as peers, each owning the math it declares. + + Args: + fragments: What each fragment is called, to the fragment: a YAML path, + YAML text, a mapping, or a loaded :class:`~math_spec.model.Spec`. + The name is what an error calls it. The order they are given in + does not reach the result. + description: What the composed model is. A fragment's own + ``description`` is about the fragment, and is not carried. + + Returns: + One mapping, ready for :func:`~math_spec.validation.to_spec`. Nothing + in it has been resolved, name-checked or lowered, and it shares no + object with any fragment. A given declaration a sibling introduces is + folded away; one nothing introduces stays under ``given:``. + + Raises: + LanguageError: Two fragments declare one name; two fragments say + different things about one dimension, relation or given + declaration; a fragment reads a name as something other than what + its sibling introduces; two fragments pin different language + versions; or their objectives run opposite ways. + FileNotFoundError: A ``str`` with no newline that names no file. + """ + read = {name: deepcopy(_declarations(fragment)) for name, fragment in fragments.items()} + merged: dict[str, Any] = {'version': _one_version(read)} + if description is not None: + merged['description'] = description + for section in SHARED_SECTIONS: + if agreed := _agreed(read, section, _singular(section)): + merged[section] = agreed + for section in OWNED_SECTIONS: + if claimed := _claimed(read, section): + merged[section] = claimed + if given := _folded(read, merged): + merged['given'] = given + if (objective := _summed_objective(read)) is not None: + merged['objective'] = objective + return merged + + +def _one_version(read: Mapping[str, dict[str, Any]]) -> int: + """The language version every fragment is written against, a fragment saying nothing being version 0.""" + declared = {name: sections.get('version', 0) for name, sections in read.items()} + if len(set(declared.values())) > 1: + spelled = ', '.join(f"'{name}' says {version}" for name, version in sorted(declared.items())) + raise LanguageError( + f'the fragments are written against different language versions: {spelled}. One model has ' + f'one version, so write the same one in each. A fragment that declares none is version 0.' + ) + return next(iter(declared.values()), 0) + + +def _author_of(read: Mapping[str, dict[str, Any]], section: str, key: str) -> str: + """The first fragment declaring *key* under *section*, for a message that names both sides.""" + return next(name for name, sections in read.items() if key in (sections.get(section) or {})) + + +def _agreed(read: Mapping[str, dict[str, Any]], section: str, label: str) -> dict[str, Any]: + """One block every fragment may declare, peers that say the same thing folded together. + + Equality of the claims rather than "the same or less": between peers + neither declaration is the one being restated, so a field only one of them + writes is a difference nothing settles. + """ + merged: dict[str, Any] = {} + for name, sections in read.items(): + for key, block in (sections.get(section) or {}).items(): + if key in merged and _claims(merged[key]) != _claims(block): + raise LanguageError( + f"fragments '{_author_of(read, section, key)}' and '{name}' say different things about " + f'the {label} {key!r}: {merged[key]!r} against {block!r}. A declaration two fragments ' + f'share is one both say the same thing about: make the two identical, or give one of ' + f'them a name of its own.' + ) + merged.setdefault(key, block) + return merged + + +def _claims(block: Any) -> Any: + """*block* without its prose, which is what the declaration says rather than a remark about it.""" + return {key: value for key, value in block.items() if key != 'description'} if isinstance(block, dict) else block + + +def _claimed(read: Mapping[str, dict[str, Any]], section: str) -> dict[str, Any]: + """One block of owned declarations, a name claimed twice being the refusal.""" + merged: dict[str, Any] = {} + for name, sections in read.items(): + for key, block in (sections.get(section) or {}).items(): + if key in merged: + raise LanguageError( + f"fragments '{_author_of(read, section, key)}' and '{name}' both declare the " + f'{_singular(section)} {key!r}. Two of the same kind of thing are two rows of a dimension ' + f'rather than two fragments: merge the fragment once, and let the data carry both. ' + f'Different math under one spelling is a rename: call one of them something else.' + ) + merged[key] = block + return merged + + +def _folded(read: Mapping[str, dict[str, Any]], merged: Mapping[str, Any]) -> dict[str, Any]: + """The ``given:`` block the composition still carries, once every reading a sibling introduces is spent. + + A given declaration is what a fragment expects of a name a sibling owns. + Where the sibling is in the composition the expectation is checked and + then dropped, so the composed model declares the name once. + """ + asked = {name: sections.get('given') or {} for name, sections in read.items()} + left: dict[str, Any] = {} + for kind, label in GIVEN_KINDS.items(): + cls = _entry_class(GivenBlock, kind) + introduced = merged.get(kind) or {} + agreed = _agreed(asked, kind, label) + for key, block in agreed.items(): + if key in introduced and not _says_less(cls, block, introduced[key]): + raise LanguageError( + f"fragment '{_author_of(asked, kind, key)}' reads the {label} {key!r} as {block!r}, where " + f"'{_author_of(read, kind, key)}' introduces it as {introduced[key]!r}. A given declaration " + f'says the same as the declaration it is folded into, or less: restate the frame as the ' + f'introducer declares it, or leave the field out.' + ) + if remaining := {key: block for key, block in agreed.items() if key not in introduced}: + left[kind] = remaining + return left + + +def _says_less(cls: type[BaseModel], reader: Any, introducer: Any) -> bool: + """Whether every claim *reader* makes is one *introducer* makes too, a field left to its default counting as said.""" + fields = cls.model_fields + return all( + introducer.get(key, fields[key].default if key in fields else None) == value + for key, value in _claims(reader).items() + ) + + +def _summed_objective(read: Mapping[str, dict[str, Any]]) -> dict[str, Any] | None: + """Every fragment's objective summed, each term in parentheses, or ``None`` where none declares one. + + The senses have to agree: a sum has one sense, and negating the odd one + out would be this function deciding what a model means. + """ + declared = {name: sections['objective'] for name, sections in read.items() if sections.get('objective')} + if not declared: + return None + senses = {name: objective.get('sense', 'minimize') for name, objective in declared.items()} + if len(set(senses.values())) > 1: + spelled = ', '.join(f"'{name}' {sense}s" for name, sense in sorted(senses.items())) + raise LanguageError( + f'the fragments disagree about which way the objective runs: {spelled}. A composed model has ' + f'one objective and one sense, so write every fragment against the same one: negate the terms ' + f'of the odd one out rather than its sense.' + ) + terms = [objective['expression'] for objective in declared.values()] + joined = terms[0] if len(terms) == 1 else ' + '.join(f'({term})' for term in terms) + return {'sense': next(iter(senses.values())), 'expression': joined} + + def override( base: str | Path | dict[str, Any] | Spec, patches: Mapping[str, str | Path | dict[str, Any] | Spec], diff --git a/tests/test_composition.py b/tests/test_composition.py index c872fbf8..2c68b827 100644 --- a/tests/test_composition.py +++ b/tests/test_composition.py @@ -2,9 +2,11 @@ # # SPDX-License-Identifier: MIT -"""`override` lays a base and its patches, and what it refuses. +"""Two verbs, and what each one refuses. -A name the patch declares is the point, and what is pinned for it is the +`merge` composes peers, so a name two fragments declare is a collision and the +order they are given in means nothing. `override` lays a base and its patches, +so a name the patch declares is the point, and what is pinned for it is the opposite: every collision the caller did not ask for is an error naming both sides. A patch that lands on nothing, two patches writing one field, a dimension redeclared or removed under the expressions written over it, and a @@ -18,9 +20,157 @@ import pytest -from math_spec import LanguageError, override, to_markdown, to_spec +from math_spec import LanguageError, merge, override, to_markdown, to_spec from tests.fixtures import DISPATCH_MODEL +#: The coupling surface a component library agrees on: one flow per port, and +#: one balance per bus. The two fragments below name `flow` and declare none of +#: it, which is what makes each of them a load error on its own. +SURFACE = { + 'dimensions': {'snapshot': {'dtype': 'int'}, 'port': {'dtype': 'str'}, 'bus': {'dtype': 'str'}}, + 'relations': {'port_bus': {'key': 'port', 'values': 'bus'}}, + 'variables': {'flow': {'dims': ['snapshot', 'port']}}, + 'constraints': { + 'balance': {'dims': ['snapshot', 'bus'], 'expression': 'sum(flow, by=port_bus, over=port, into=bus) == 0'} + }, +} + +SUPPLY = { + 'dimensions': {'snapshot': {'dtype': 'int'}, 'port': {'dtype': 'str'}, 'generator': {'dtype': 'str'}}, + 'relations': {'gen_port': {'key': 'generator', 'values': 'port'}}, + 'parameters': {'gen_cost': {'dims': ['generator']}, 'gen_p_max': {'dims': ['generator']}}, + 'variables': {'gen_p': {'dims': ['snapshot', 'generator'], 'bounds': {'lower': 0, 'upper': 'gen_p_max'}}}, + 'constraints': { + 'gen_injects': { + 'dims': ['snapshot', 'generator'], + 'expression': 'at(flow, by=gen_port, over=port, into=generator) == gen_p', + } + }, + 'objective': {'sense': 'minimize', 'expression': 'sum(gen_p * gen_cost)'}, +} + +DEMAND = { + 'dimensions': {'snapshot': {'dtype': 'int'}, 'port': {'dtype': 'str'}, 'demand': {'dtype': 'str'}}, + 'relations': {'dem_port': {'key': 'demand', 'values': 'port'}}, + 'parameters': {'dem_load': {'dims': ['snapshot', 'demand']}}, + 'constraints': { + 'dem_withdraws': { + 'dims': ['snapshot', 'demand'], + 'expression': 'at(flow, by=dem_port, over=port, into=demand) == -dem_load', + } + }, +} + +LIBRARY = {'surface': SURFACE, 'supply': SUPPLY, 'demand': DEMAND} + + +def test_a_fragment_names_what_a_sibling_declares(): + """The whole reason merging happens before validation: `supply` reads `flow` and declares none of it.""" + with pytest.raises(LanguageError, match=r'flow'): + to_spec(SUPPLY) + spec = to_spec(merge(LIBRARY)) + assert sorted(spec.variables) == ['flow', 'gen_p'], "both fragments' columns are in the one model" + assert to_markdown(spec), 'a composed library prints as math' + + +def test_the_balance_does_not_grow_when_a_component_type_is_added(): + """What the port convention buys: a component pins its own flow rather than adding a term.""" + three = to_spec(merge(LIBRARY)).constraints['balance'].expression + storage = { + 'dimensions': {'snapshot': {'dtype': 'int'}, 'port': {'dtype': 'str'}, 'store': {'dtype': 'str'}}, + 'relations': {'st_port': {'key': 'store', 'values': 'port'}}, + 'parameters': {'st_capacity': {'dims': ['store']}}, + 'variables': {'st_p': {'dims': ['snapshot', 'store'], 'bounds': {'lower': 0, 'upper': 'st_capacity'}}}, + 'constraints': { + 'st_injects': { + 'dims': ['snapshot', 'store'], + 'expression': 'at(flow, by=st_port, over=port, into=store) == st_p', + } + }, + } + four = to_spec(merge({**LIBRARY, 'storage': storage})).constraints['balance'].expression + assert three == four, 'the balance is written once, whatever is plugged into it' + + +def test_merging_is_order_independent(): + assert merge(LIBRARY) == merge(dict(reversed(list(LIBRARY.items())))) + + +def test_the_fragments_are_never_mutated_and_share_nothing_with_the_result(): + before = copy.deepcopy(LIBRARY) + composed = merge(LIBRARY) + assert before == LIBRARY, 'a composed model is a new mapping, and the fragments are untouched' + assert composed['parameters']['gen_cost'] is not SUPPLY['parameters']['gen_cost'], ( + "a declaration carried over is a copy, not the fragment's own object" + ) + + +def test_a_name_two_fragments_declare_is_refused(): + twin = {**DEMAND, 'parameters': {'gen_cost': {'dims': ['demand']}}} + with pytest.raises(LanguageError) as raised: + merge({'supply': SUPPLY, 'twin': twin}) + message = str(raised.value) + assert "'supply'" in message and "'twin'" in message, 'a collision names both fragments' + assert 'two rows of a dimension' in message, 'the message names the rewrite it usually is' + + +def test_a_dimension_two_fragments_describe_differently_is_refused(): + relabelled = {**DEMAND, 'dimensions': {**DEMAND['dimensions'], 'snapshot': {'dtype': 'str'}}} + with pytest.raises(LanguageError, match=r'say different things about the dimension') as raised: + merge({'supply': SUPPLY, 'demand': relabelled}) + assert 'a name of its own' in str(raised.value), 'the refusal names the rewrite' + + +def test_two_descriptions_of_one_dimension_agree_and_the_first_is_carried(): + """Prose is not a claim, so two fragments describing one dimension in their own words agree about it.""" + first = {**SUPPLY, 'dimensions': {**SUPPLY['dimensions'], 'snapshot': {'dtype': 'int', 'description': 'an hour'}}} + second = {**DEMAND, 'dimensions': {**DEMAND['dimensions'], 'snapshot': {'dtype': 'int', 'description': 'a step'}}} + composed = merge({'supply': first, 'demand': second}) + assert composed['dimensions']['snapshot'] == {'dtype': 'int', 'description': 'an hour'} + + +def test_the_objectives_are_summed_each_term_parenthesised(): + """`a + b * k` reassociates, so an unparenthesised join composes a different objective.""" + priced = {**DEMAND, 'objective': {'sense': 'minimize', 'expression': 'sum(dem_load) * 2'}} + composed = merge({'surface': SURFACE, 'supply': SUPPLY, 'demand': priced}) + assert composed['objective']['expression'] == '(sum(gen_p * gen_cost)) + (sum(dem_load) * 2)' + + +def test_one_fragment_s_objective_is_carried_as_it_was_written(): + assert merge(LIBRARY)['objective']['expression'] == SUPPLY['objective']['expression'] + + +def test_objectives_that_run_opposite_ways_are_refused(): + maximised = {**DEMAND, 'objective': {'sense': 'maximize', 'expression': 'sum(dem_load)'}} + with pytest.raises(LanguageError, match=r'which way the objective runs'): + merge({'supply': SUPPLY, 'demand': maximised}) + + +def test_fragments_pinning_different_versions_are_refused(): + with pytest.raises(LanguageError, match=r'different language versions'): + merge({'supply': {**SUPPLY, 'version': 0}, 'demand': {**DEMAND, 'version': 1}}) + + +def test_the_description_belongs_to_the_composition(): + described = merge({**LIBRARY, 'supply': {**SUPPLY, 'description': 'a fleet'}}, description='a fleet against a load') + assert described['description'] == 'a fleet against a load' + assert 'description' not in merge({**LIBRARY, 'supply': {**SUPPLY, 'description': 'a fleet'}}), ( + "no fragment's own description is carried" + ) + + +def test_merge_composes_the_model_and_override_configures_the_run(): + """The two verbs meet by taking and returning what the other does.""" + run = override(merge(LIBRARY), {'project': {'variables': {'gen_p': {'where': 'gen_p_max > 0'}}}}) + assert to_spec(run).variables['gen_p'].where == 'gen_p_max > 0' + + +def test_a_fragment_is_a_path_as_readily_as_a_mapping(tmp_path): + surface = tmp_path / 'surface.yaml' + surface.write_text(to_spec(SURFACE).to_yaml(), encoding='utf-8') + assert to_spec(merge({**LIBRARY, 'surface': str(surface)})) == to_spec(merge(LIBRARY)) + + #: A patch that adds what it needs and a constraint that reads it, so the #: composed model is one `to_spec` accepts rather than only one that lays. CARBON = { diff --git a/tests/test_given.py b/tests/test_given.py index 02ccf066..01890dec 100644 --- a/tests/test_given.py +++ b/tests/test_given.py @@ -4,17 +4,19 @@ """What a file reads and does not build: a column, and a row family. -A fragment reads a column the file beside it introduces. A layer reads a -column, or the dual of a row family, that a model outside the language holds. -What both need is that the file stands on its own: it loads, it lowers, and it -prints as math, without the thing that owns what it reads. +A fragment reads a column the file beside it introduces, and `merge` folds the +two together, so the composed model carries no trace of the reading. A layer +reads a column, or the dual of a row family, that a model outside the language +holds, so there is nothing to fold into and the program carries the name for a +consumer to bind. What both need is that the file stands on its own: it loads, +it lowers, and it prints as math, without the thing that owns what it reads. """ from __future__ import annotations import pytest -from math_spec import FORMATS, LanguageError, advice, to_markdown, to_program, to_spec, typeset +from math_spec import FORMATS, LanguageError, advice, merge, to_markdown, to_program, to_spec, typeset #: One component file: it pins the flow at its own port, and the column it #: pins belongs to another fragment. @@ -34,6 +36,16 @@ 'objective': {'sense': 'minimize', 'expression': 'sum(gen_p * gen_cost)'}, } +#: The fragment that introduces `flow`, with the bounds and the balance that go with it. +SURFACE = { + 'dimensions': {'snapshot': {'dtype': 'int'}, 'port': {'dtype': 'str'}, 'bus': {'dtype': 'str'}}, + 'relations': {'port_bus': {'key': 'port', 'values': 'bus'}}, + 'variables': {'flow': {'dims': ['snapshot', 'port'], 'bounds': {'lower': -1000, 'upper': 1000}}}, + 'constraints': { + 'balance': {'dims': ['snapshot', 'bus'], 'expression': 'sum(flow, by=port_bus, over=port, into=bus) == 0'} + }, +} + def test_given_holds_two_kinds_and_refuses_a_third(): """The section is closed, so a kind nobody has admitted yet is the schema's own refusal.""" @@ -127,6 +139,59 @@ def test_an_expression_reads_a_given_column_as_it_reads_any_other(): assert spec.constraints['gen_injects'].dims == ['snapshot', 'generator'] +def test_merging_folds_the_given_declaration_into_the_one_that_introduces_it(): + composed = merge({'surface': SURFACE, 'supply': SUPPLY}) + assert 'given' not in composed, 'the expectation is spent once the column is in the composition' + spec = to_spec(composed) + assert sorted(spec.variables) == ['flow', 'gen_p'] + assert spec.variables['flow'].bounds.lower == -1000, "the introducer's declaration is the one that survives" + assert sorted(to_program(spec).variables) == ['flow', 'gen_p'], 'a composed library lowers like any model' + + +@pytest.mark.parametrize( + 'reads', + [ + pytest.param({'dims': ['snapshot', 'port']}, id='the-frame-alone'), + pytest.param({'dims': ['snapshot', 'port'], 'domain': 'continuous'}, id='the-domain-the-introducer-defaults'), + pytest.param({'dims': ['snapshot', 'port'], 'description': 'the flow, in my words'}, id='its-own-prose'), + ], +) +def test_a_given_declaration_may_say_less_than_the_introducer(reads): + """Bounds are the introducer's, so the reader states the frame and stops.""" + composed = merge({'surface': SURFACE, 'supply': {**SUPPLY, 'given': {'variables': {'flow': reads}}}}) + assert to_spec(composed).variables['flow'].bounds.upper == 1000 + + +@pytest.mark.parametrize( + 'reads', + [ + pytest.param({'dims': ['snapshot', 'generator']}, id='another-frame'), + pytest.param({'dims': ['snapshot', 'port'], 'domain': 'binary'}, id='another-domain'), + ], +) +def test_a_given_declaration_that_disagrees_with_the_introducer_is_refused(reads): + misread = {**SUPPLY, 'given': {'variables': {'flow': reads}}} + with pytest.raises(LanguageError, match=r'says the same as the declaration it is folded into, or less') as raised: + merge({'surface': SURFACE, 'supply': misread}) + message = str(raised.value) + assert "'supply'" in message and "'surface'" in message, 'both sides of a disagreement are named' + + +def test_two_fragments_must_read_one_column_the_same_way(): + other = { + 'dimensions': {'snapshot': {'dtype': 'int'}, 'port': {'dtype': 'str'}}, + 'given': {'variables': {'flow': {'dims': ['port']}}}, + } + with pytest.raises(LanguageError, match=r'say different things about the given variable'): + merge({'supply': SUPPLY, 'other': other}) + + +def test_a_given_declaration_nothing_introduces_stays_for_a_consumer_to_bind(): + composed = merge({'supply': SUPPLY, 'other': {'dimensions': {'snapshot': {'dtype': 'int'}}}}) + assert composed['given'] == SUPPLY['given'], 'a name no fragment introduces is still read, and is carried' + assert sorted(to_program(composed).given.variables) == ['flow'] + + #: A layer over a model this language never sees: it reads a column and the #: dual of a row family, and adds one constraint of its own. LAYER = { @@ -194,6 +259,19 @@ def test_a_given_row_family_is_refused_where_it_oversteps(block, says): assert says in str(raised.value) +def test_merging_folds_a_row_family_into_the_file_that_builds_it(): + builder = { + 'dimensions': {'snapshot': {'dtype': 'int'}, 'bus': {'dtype': 'str'}}, + 'variables': {'p': {'dims': ['snapshot', 'bus'], 'bounds': {'lower': 0}}}, + 'constraints': {'balance': {'dims': ['snapshot', 'bus'], 'expression': 'p >= 0'}}, + } + composed = merge({'builder': builder, 'layer': LAYER}) + assert 'given' not in composed + program = to_program(composed) + assert sorted(program.constraints) == ['balance', 'cap'] + assert not program.given.constraints, 'nothing is left for a consumer to bind' + + def test_the_advice_names_every_declaration_a_consumer_has_to_bind(): subjects = {note.subject for note in advice(LAYER) if note.kind == 'given'} assert subjects == {'p', 'balance'}, 'both the column and the row family are named' diff --git a/tests/test_public_surface.py b/tests/test_public_surface.py index 03e4356b..811e3c24 100644 --- a/tests/test_public_surface.py +++ b/tests/test_public_surface.py @@ -25,8 +25,8 @@ { # the two public states, and the conversion to each 'Spec', 'to_spec', 'program', 'to_program', - # the file-level verb that lays patches over a base - 'override', + # the two file-level verbs: peers composed, and patches laid over a base + 'merge', 'override', # the error tree 'MathSpecError', 'LanguageError', 'SchemaError', 'DimensionError', 'PiecewiseExpansionError', 'did_you_mean', 'schema_error', From 1da63b5d5ab655c50f50c698e41fabd6b6b209b7 Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 17:35:17 +0200 Subject: [PATCH 6/9] fix(language): a merged objective sums in one order, and a fragment cannot read what it builds The composed objective joined its terms in fragment-iteration order, so the same fragments under two argument orders gave two expressions. The terms are summed in the fragments' name order now. A fragment that declares a name and reads it under given: too is refused, naming the fragment, the name and the rewrite. Such a file does not load on its own, and folding the reading away put it in a model that loads. merge writes version: only where a fragment declares one. Two fragments declaring different versions are still refused. --- docs/howto/compose.md | 33 ++++++++++----- src/math_spec/advice.py | 2 +- src/math_spec/composition.py | 64 +++++++++++++++++++++-------- tests/test_advice.py | 1 + tests/test_composition.py | 79 ++++++++++++++++++++++++------------ tests/test_given.py | 24 +++++++++++ 6 files changed, 150 insertions(+), 53 deletions(-) diff --git a/docs/howto/compose.md b/docs/howto/compose.md index 80c2850a..e647d2ec 100644 --- a/docs/howto/compose.md +++ b/docs/howto/compose.md @@ -96,7 +96,8 @@ file, and the two compose as `override(merge({…}), {…})`. `merge` folds each given declaration into the declaration that introduces it, so `spec` declares `Port_p` once and carries no `given:`. The objectives - of the fragments are summed, each term in parentheses. + of the fragments are summed, each term in parentheses, in the order the + fragment names sort in. 4. **Add a component type without touching the balance.** A component file pins the flow at its own port rather than adding a term to the balance, so @@ -106,16 +107,16 @@ file, and the two compose as `override(merge({…}), {…})`. ## What a fragment may share -| The entry | What happens | -| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -| a dimension or a relation | every fragment may declare it, and the ones that do say the same thing about it | -| a `description` on a shared dimension or relation | it is prose rather than a claim, and the first fragment's wording is carried | -| any other declaration | one fragment declares it, and a second is refused | -| an entry under `given: variables:` or `given: constraints:` | it is checked against the fragment that introduces the name, then folded into it | -| a given entry no fragment introduces | it stays under `given:` for a consumer to bind | -| `objective` | the terms are summed, each in parentheses, and the senses agree | -| `version` | every fragment says the same one, and a fragment that says none is version 0 | -| `description` at the top of a fragment | it is about the fragment and is not carried. Pass the composed model's as `description=` | +| The entry | What happens | +| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------- | +| a dimension or a relation | every fragment may declare it, and the ones that do say the same thing about it | +| a `description` on a shared dimension or relation | it is prose rather than a claim, and the first fragment's wording is carried | +| any other declaration | one fragment declares it, and a second is refused | +| an entry under `given: variables:` or `given: constraints:` | it is checked against the fragment that introduces the name, then folded into it | +| a given entry no fragment introduces | it stays under `given:` for a consumer to bind | +| `objective` | the terms are summed in fragment-name order, each in parentheses, and the senses agree | +| `version` | the fragments that write one say the same one, and a composition nothing pins writes none | +| `description` at the top of a fragment | it is about the fragment and is not carried. Pass the composed model's as `description=` | ## A name two fragments declare @@ -139,6 +140,16 @@ fragment 'generator' reads the given variable 'Port_p' as {'dims': ['snapshot', Two fragments that both only read a column have to read it the same way, and a difference is refused as it is for a dimension. +## A name one fragment both builds and reads + +A fragment reads what another file builds. A fragment that declares a name and +reads it as well is a file `to_spec` refuses on its own, so `merge` refuses it +too rather than folding the reading away: + +```text +fragment 'generator' declares the variable 'Generator_p' and reads it under 'given: variables:' as well. A given declaration is what one file expects of another, and this fragment builds the name itself: drop the given entry, or move the declaration to the fragment this one reads it from. +``` + ## A base and its patches 1. **Write the base as a model**, and each patch as the change it makes. A diff --git a/src/math_spec/advice.py b/src/math_spec/advice.py index f538ead4..f415edc2 100644 --- a/src/math_spec/advice.py +++ b/src/math_spec/advice.py @@ -53,7 +53,7 @@ def _given(program: Program) -> list[Advice]: name, f"{kind} '{name}' is read here and built elsewhere: a consumer binds it to the model this " f'one is layered onto, checks the frame, and refuses where it cannot bind it. A fragment is ' - f'composed instead: merge() folds it into the file that introduces it.', + f'composed instead: merge() folds this declaration into the one a sibling introduces.', ) for kind, group in (('variable', program.given.variables), ('row family', program.given.constraints)) for name in group diff --git a/src/math_spec/composition.py b/src/math_spec/composition.py index e5e649cd..574aa30c 100644 --- a/src/math_spec/composition.py +++ b/src/math_spec/composition.py @@ -19,12 +19,13 @@ descriptions of one dimension agree, and the first fragment's is carried. * **Every other declaration is owned.** A name two fragments declare is refused, both named. -* **The objectives are summed**, each term in parentheses, and the senses have - to agree. +* **The objectives are summed**, each term in parentheses, in the fragments' + name order, and the senses have to agree. * **A given declaration is folded** into the declaration that introduces the name, once the reader is checked to say the same as the introducer or less. - Two fragments that both only read a name have to read it the same way. What - no fragment introduces stays under ``given:`` for a consumer to bind. + Two fragments that both only read a name have to read it the same way, and a + fragment that declares a name and reads it as well is refused. What no + fragment introduces stays under ``given:`` for a consumer to bind. A patch says only what it changes, because declarations are laid over a field at a time:: @@ -120,12 +121,15 @@ def merge( LanguageError: Two fragments declare one name; two fragments say different things about one dimension, relation or given declaration; a fragment reads a name as something other than what - its sibling introduces; two fragments pin different language - versions; or their objectives run opposite ways. + its sibling introduces; a fragment declares a name and reads it as + well; two fragments pin different language versions; or their + objectives run opposite ways. FileNotFoundError: A ``str`` with no newline that names no file. """ read = {name: deepcopy(_declarations(fragment)) for name, fragment in fragments.items()} - merged: dict[str, Any] = {'version': _one_version(read)} + merged: dict[str, Any] = {} + if (version := _one_version(read)) is not None: + merged['version'] = version if description is not None: merged['description'] = description for section in SHARED_SECTIONS: @@ -141,16 +145,21 @@ def merge( return merged -def _one_version(read: Mapping[str, dict[str, Any]]) -> int: - """The language version every fragment is written against, a fragment saying nothing being version 0.""" - declared = {name: sections.get('version', 0) for name, sections in read.items()} +def _one_version(read: Mapping[str, dict[str, Any]]) -> int | None: + """The language version the fragments are written against, or ``None`` where none of them writes one. + + A fragment that writes none is version 0, which is the schema's default, so + a composition of such fragments claims no version rather than writing the + default out as though a file had asked for it. + """ + declared = {name: sections['version'] for name, sections in read.items() if 'version' in sections} if len(set(declared.values())) > 1: spelled = ', '.join(f"'{name}' says {version}" for name, version in sorted(declared.items())) raise LanguageError( f'the fragments are written against different language versions: {spelled}. One model has ' - f'one version, so write the same one in each. A fragment that declares none is version 0.' + f'one version, so write the same one in each, or leave it out of the fragments that do not pin it.' ) - return next(iter(declared.values()), 0) + return next(iter(declared.values()), None) def _author_of(read: Mapping[str, dict[str, Any]], section: str, key: str) -> str: @@ -208,6 +217,7 @@ def _folded(read: Mapping[str, dict[str, Any]], merged: Mapping[str, Any]) -> di then dropped, so the composed model declares the name once. """ asked = {name: sections.get('given') or {} for name, sections in read.items()} + _reads_only_what_it_does_not_build(read, asked) left: dict[str, Any] = {} for kind, label in GIVEN_KINDS.items(): cls = _entry_class(GivenBlock, kind) @@ -226,6 +236,26 @@ def _folded(read: Mapping[str, dict[str, Any]], merged: Mapping[str, Any]) -> di return left +def _reads_only_what_it_does_not_build(read: Mapping[str, dict[str, Any]], asked: Mapping[str, dict[str, Any]]) -> None: + """Refuse a fragment that declares a name and reads it under ``given:`` too. + + :func:`~math_spec.validation.to_spec` refuses such a file, so folding the + reading away silently would put a fragment that loads nowhere on its own + into a composition that loads. + """ + for name, given in asked.items(): + for kind in GIVEN_KINDS: + built = read[name].get(kind) or {} + for key in given.get(kind) or {}: + if key in built: + raise LanguageError( + f"fragment '{name}' declares the {_singular(kind)} {key!r} and reads it under " + f"'given: {kind}:' as well. A given declaration is what one file expects of another, " + f'and this fragment builds the name itself: drop the given entry, or move the ' + f'declaration to the fragment this one reads it from.' + ) + + def _says_less(cls: type[BaseModel], reader: Any, introducer: Any) -> bool: """Whether every claim *reader* makes is one *introducer* makes too, a field left to its default counting as said.""" fields = cls.model_fields @@ -238,8 +268,10 @@ def _says_less(cls: type[BaseModel], reader: Any, introducer: Any) -> bool: def _summed_objective(read: Mapping[str, dict[str, Any]]) -> dict[str, Any] | None: """Every fragment's objective summed, each term in parentheses, or ``None`` where none declares one. - The senses have to agree: a sum has one sense, and negating the odd one - out would be this function deciding what a model means. + The terms are summed in the fragments' name order, so the order they were + passed in does not reach the expression. The senses have to agree: a sum has + one sense, and negating the odd one out would be this function deciding what + a model means. """ declared = {name: sections['objective'] for name, sections in read.items() if sections.get('objective')} if not declared: @@ -252,7 +284,7 @@ def _summed_objective(read: Mapping[str, dict[str, Any]]) -> dict[str, Any] | No f'one objective and one sense, so write every fragment against the same one: negate the terms ' f'of the odd one out rather than its sense.' ) - terms = [objective['expression'] for objective in declared.values()] + terms = [objective['expression'] for _, objective in sorted(declared.items())] joined = terms[0] if len(terms) == 1 else ' + '.join(f'({term})' for term in terms) return {'sense': next(iter(senses.values())), 'expression': joined} @@ -292,7 +324,7 @@ def override( def _declarations(source: str | Path | dict[str, Any] | Spec) -> dict[str, Any]: - """A base or a patch as the mapping it declares, whatever shape it arrived in. + """A fragment, a base or a patch as the mapping it declares, whatever shape it arrived in. Deliberately not :func:`~math_spec.validation.to_spec`: a patch carrying a ``null`` or naming only the field it changes is not a model, and validating diff --git a/tests/test_advice.py b/tests/test_advice.py index 1a5e4d64..2105c5ef 100644 --- a/tests/test_advice.py +++ b/tests/test_advice.py @@ -83,6 +83,7 @@ def test_a_column_read_and_not_built_is_advised(): (note,) = advice(READS_A_COLUMN) assert (note.kind, note.subject) == ('given', 'flow') assert 'binds it to the model' in str(note), 'the note says whose job the column is' + assert 'merge()' in str(note), 'and names the verb that folds the reading away where a sibling builds it' def test_every_kind_a_consumer_can_pin_against_is_produced_here(): diff --git a/tests/test_composition.py b/tests/test_composition.py index 2c68b827..f13d8f41 100644 --- a/tests/test_composition.py +++ b/tests/test_composition.py @@ -92,8 +92,18 @@ def test_the_balance_does_not_grow_when_a_component_type_is_added(): assert three == four, 'the balance is written once, whatever is plugged into it' -def test_merging_is_order_independent(): - assert merge(LIBRARY) == merge(dict(reversed(list(LIBRARY.items())))) +@pytest.mark.parametrize( + 'fragments', + [ + pytest.param(LIBRARY, id='one-objective'), + pytest.param( + {**LIBRARY, 'demand': {**DEMAND, 'objective': {'sense': 'minimize', 'expression': 'sum(dem_load)'}}}, + id='an-objective-in-two-fragments', + ), + ], +) +def test_merging_is_order_independent(fragments): + assert merge(fragments) == merge(dict(reversed(list(fragments.items())))) def test_the_fragments_are_never_mutated_and_share_nothing_with_the_result(): @@ -105,20 +115,41 @@ def test_the_fragments_are_never_mutated_and_share_nothing_with_the_result(): ) -def test_a_name_two_fragments_declare_is_refused(): - twin = {**DEMAND, 'parameters': {'gen_cost': {'dims': ['demand']}}} +@pytest.mark.parametrize( + ('fragments', 'says'), + [ + pytest.param( + {'supply': SUPPLY, 'demand': {**DEMAND, 'parameters': {'gen_cost': {'dims': ['demand']}}}}, + 'two rows of a dimension', + id='one-name-declared-twice', + ), + pytest.param( + { + 'supply': SUPPLY, + 'demand': {**DEMAND, 'dimensions': {**DEMAND['dimensions'], 'snapshot': {'dtype': 'str'}}}, + }, + 'give one of them a name of its own', + id='one-dimension-described-two-ways', + ), + pytest.param( + {'supply': SUPPLY, 'demand': {**DEMAND, 'objective': {'sense': 'maximize', 'expression': 'sum(dem_load)'}}}, + 'negate the terms', + id='objectives-that-run-opposite-ways', + ), + pytest.param( + {'supply': {**SUPPLY, 'version': 0}, 'demand': {**DEMAND, 'version': 1}}, + 'One model has one version', + id='two-language-versions', + ), + ], +) +def test_a_disagreement_between_fragments_is_refused(fragments, says): + """No order of the fragments settles any of these, so each is a refusal rather than a rule.""" with pytest.raises(LanguageError) as raised: - merge({'supply': SUPPLY, 'twin': twin}) + merge(fragments) message = str(raised.value) - assert "'supply'" in message and "'twin'" in message, 'a collision names both fragments' - assert 'two rows of a dimension' in message, 'the message names the rewrite it usually is' - - -def test_a_dimension_two_fragments_describe_differently_is_refused(): - relabelled = {**DEMAND, 'dimensions': {**DEMAND['dimensions'], 'snapshot': {'dtype': 'str'}}} - with pytest.raises(LanguageError, match=r'say different things about the dimension') as raised: - merge({'supply': SUPPLY, 'demand': relabelled}) - assert 'a name of its own' in str(raised.value), 'the refusal names the rewrite' + assert says in message, 'the refusal names the rewrite rather than only what is wrong' + assert all(f"'{name}'" in message for name in fragments), 'a disagreement names both fragments' def test_two_descriptions_of_one_dimension_agree_and_the_first_is_carried(): @@ -126,29 +157,27 @@ def test_two_descriptions_of_one_dimension_agree_and_the_first_is_carried(): first = {**SUPPLY, 'dimensions': {**SUPPLY['dimensions'], 'snapshot': {'dtype': 'int', 'description': 'an hour'}}} second = {**DEMAND, 'dimensions': {**DEMAND['dimensions'], 'snapshot': {'dtype': 'int', 'description': 'a step'}}} composed = merge({'supply': first, 'demand': second}) - assert composed['dimensions']['snapshot'] == {'dtype': 'int', 'description': 'an hour'} + assert composed['dimensions']['snapshot'] == {'dtype': 'int', 'description': 'an hour'}, ( + "the claim is carried whole, under the first fragment's wording of the prose" + ) def test_the_objectives_are_summed_each_term_parenthesised(): """`a + b * k` reassociates, so an unparenthesised join composes a different objective.""" priced = {**DEMAND, 'objective': {'sense': 'minimize', 'expression': 'sum(dem_load) * 2'}} composed = merge({'surface': SURFACE, 'supply': SUPPLY, 'demand': priced}) - assert composed['objective']['expression'] == '(sum(gen_p * gen_cost)) + (sum(dem_load) * 2)' + assert composed['objective']['expression'] == '(sum(dem_load) * 2) + (sum(gen_p * gen_cost))', ( + "the terms are summed in the fragments' name order, which no argument order can change" + ) def test_one_fragment_s_objective_is_carried_as_it_was_written(): assert merge(LIBRARY)['objective']['expression'] == SUPPLY['objective']['expression'] -def test_objectives_that_run_opposite_ways_are_refused(): - maximised = {**DEMAND, 'objective': {'sense': 'maximize', 'expression': 'sum(dem_load)'}} - with pytest.raises(LanguageError, match=r'which way the objective runs'): - merge({'supply': SUPPLY, 'demand': maximised}) - - -def test_fragments_pinning_different_versions_are_refused(): - with pytest.raises(LanguageError, match=r'different language versions'): - merge({'supply': {**SUPPLY, 'version': 0}, 'demand': {**DEMAND, 'version': 1}}) +def test_a_version_no_fragment_pins_is_left_out(): + assert 'version' not in merge(LIBRARY), 'a composition claims a version only where a fragment wrote one' + assert merge({**LIBRARY, 'supply': {**SUPPLY, 'version': 0}})['version'] == 0 def test_the_description_belongs_to_the_composition(): diff --git a/tests/test_given.py b/tests/test_given.py index 01890dec..f80b8857 100644 --- a/tests/test_given.py +++ b/tests/test_given.py @@ -186,6 +186,30 @@ def test_two_fragments_must_read_one_column_the_same_way(): merge({'supply': SUPPLY, 'other': other}) +@pytest.mark.parametrize( + ('fragment', 'says'), + [ + pytest.param( + {**SUPPLY, 'given': {'variables': {'gen_p': {'dims': ['snapshot', 'generator']}}}}, + "the variable 'gen_p'", + id='a-column-it-builds', + ), + pytest.param( + {**SUPPLY, 'given': {'constraints': {'gen_injects': {'dims': ['snapshot', 'generator']}}}}, + "the constraint 'gen_injects'", + id='a-row-family-it-builds', + ), + ], +) +def test_a_fragment_that_reads_what_it_builds_is_refused(fragment, says): + """`to_spec` refuses such a file, and folding it silently would put it in a model that loads.""" + with pytest.raises(LanguageError) as raised: + merge({'surface': SURFACE, 'supply': fragment}) + message = str(raised.value) + assert says in message and "'supply'" in message, 'the refusal names the fragment and the name it reads twice' + assert 'drop the given' in message, 'the refusal names the rewrite' + + def test_a_given_declaration_nothing_introduces_stays_for_a_consumer_to_bind(): composed = merge({'supply': SUPPLY, 'other': {'dimensions': {'snapshot': {'dtype': 'int'}}}}) assert composed['given'] == SUPPLY['given'], 'a name no fragment introduces is still read, and is carried' From cea9efe6ffa1c740b14dec762a834aca49055307 Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 19:25:44 +0200 Subject: [PATCH 7/9] docs(examples): a component library composes into one model, and each file prints alone examples/library/ holds a coupling surface, a generator and a load as fragments, and variants/commitment.yaml as a patch over their composition. One symbol table serves the library, cut per page to what each model declares. tools/gallery.py prints a page per fragment and a composed page with one tab per variant; tab() moves to tools/_page.py; render_tex skips variants/. tests/test_library_example.py holds the library to what its pages claim. Sentence lengths on the five pages: median 8 to 15 words, three sentences over 25. --- .prettierignore | 4 + docs/examples/index.md | 3 + docs/examples/library/composed.md | 261 ++++++++++++++++++++++ docs/examples/library/generator.md | 114 ++++++++++ docs/examples/library/index.md | 66 ++++++ docs/examples/library/load.md | 69 ++++++ docs/examples/library/surface.md | 77 +++++++ examples/library/generator.yaml | 37 +++ examples/library/load.yaml | 25 +++ examples/library/surface.yaml | 24 ++ examples/library/variants/commitment.yaml | 21 ++ examples/symbols/library.yaml | 24 ++ mkdocs.yml | 6 + tests/test_library_example.py | 102 +++++++++ tools/_page.py | 6 + tools/gallery.py | 77 ++++++- tools/home_math.py | 9 +- tools/render_tex.py | 6 +- 18 files changed, 917 insertions(+), 14 deletions(-) create mode 100644 docs/examples/library/composed.md create mode 100644 docs/examples/library/generator.md create mode 100644 docs/examples/library/index.md create mode 100644 docs/examples/library/load.md create mode 100644 docs/examples/library/surface.md create mode 100644 examples/library/generator.yaml create mode 100644 examples/library/load.yaml create mode 100644 examples/library/surface.yaml create mode 100644 examples/library/variants/commitment.yaml create mode 100644 examples/symbols/library.yaml create mode 100644 tests/test_library_example.py diff --git a/.prettierignore b/.prettierignore index 66add10f..ce50c8b3 100644 --- a/.prettierignore +++ b/.prettierignore @@ -29,6 +29,10 @@ docs/examples/pypsa_linearized_uc.md docs/examples/pypsa_losses.md docs/examples/pypsa_stochastic.md docs/examples/pypsa_multi_period.md +docs/examples/library/surface.md +docs/examples/library/generator.md +docs/examples/library/load.md +docs/examples/library/composed.md # The PyPSA reference scripts write this file; prettier would reformat what # they stamp, and the two would fight over it exactly as above. diff --git a/docs/examples/index.md b/docs/examples/index.md index e446a64c..b72748c4 100644 --- a/docs/examples/index.md +++ b/docs/examples/index.md @@ -17,6 +17,9 @@ Every model is a file under `examples/` in the repository. - [PyPSA in one file](pypsa.md) states the model `n.optimize()` builds, one declaration at a time. PyPSA's name for each row sits beside the YAML and the equation. +- [A component library](library/index.md) is several files that compose into + one model. Each file reads the coupling surface and prints on its own, and the + composed page shows what `merge` returns. The math on these pages is printed by the typesetter from the file above it. See [Typeset the math](../reference/typeset.md) to print your own. diff --git a/docs/examples/library/composed.md b/docs/examples/library/composed.md new file mode 100644 index 00000000..188dd10f --- /dev/null +++ b/docs/examples/library/composed.md @@ -0,0 +1,261 @@ + + +# The composed model + +What [the surface](surface.md), [generators](generator.md) and +[loads](load.md) make together: + +```python +import math_spec as ms + +model = ms.merge({'surface': 'surface.yaml', 'generator': 'generator.yaml', 'load': 'load.yaml'}) +spec = ms.to_spec(model) +``` + +The file below is `model`, the mapping `merge` returns, written as YAML. No +fragment holds it, and nothing in the repository commits it. `Port_p` is one +declaration here. Each component fragment read it under `given:`, and merging +folded those readings into the surface's own declaration. + +The objective is the generator's, carried as it was written, since no other +fragment prices anything. A second priced fragment would add its term to this +one, each term in parentheses. + +The math under the file has a tab per formulation. **As composed** is the model +above. **With commitment** lays `variants/commitment.yaml` over it with +[`override`](../../howto/compose.md#a-base-and-its-patches), which makes the +generator a committed unit: + +```python +spec = ms.to_spec(ms.override(model, {'commitment': 'variants/commitment.yaml'})) +``` + +A patch is refused on its own, since it edits declarations it does not +declare. So the model it lands on is the only place its math exists, and the +tab prints the patch beside that math. + + +```yaml +dimensions: + snapshot: {dtype: datetime, description: dispatch periods} + bus: {dtype: str, description: network nodes} + port: {dtype: str, description: 'the connections components make, one label per connection'} + generator: {dtype: str, description: 'generating units, each on one port'} + load: {dtype: str, description: 'demands, each on one port'} +relations: + Port_bus: {key: port, values: bus} + Generator_port: {key: generator, values: port} + Load_port: {key: load, values: port} +parameters: + Generator_p_nom: + dims: [generator] + description: nominal power + Generator_marginal_cost: + dims: [generator] + description: cost of one unit of output + Load_p_set: + dims: [snapshot, load] + description: '`Load-p_set` — what a load takes in a snapshot' +variables: + Port_p: + dims: [snapshot, port] + description: what a port puts into its bus in a snapshot, negative for a withdrawal + Generator_p: + dims: [snapshot, generator] + bounds: {lower: 0, upper: Generator_p_nom} + description: '`Generator-p` — what a generator produces in a snapshot' +constraints: + Bus_nodal_balance: + description: '`Bus-nodal_balance` — what the ports on a bus put in nets to nothing' + dims: [snapshot, bus] + expression: sum(Port_p, by=Port_bus, over=port, into=bus) == 0 + Generator_injection: + description: 'what a generator produces is what its port injects. No PyPSA row stands for this: PyPSA + writes the generator into the balance instead' + dims: [snapshot, generator] + expression: at(Port_p, by=Generator_port, over=port, into=generator) == Generator_p + Load_withdrawal: + description: 'what a load takes is what its port withdraws. No PyPSA row stands for this: PyPSA writes + the load into the balance instead' + dims: [snapshot, load] + expression: at(Port_p, by=Load_port, over=port, into=load) == -Load_p_set +objective: {sense: minimize, expression: sum(Generator_p * Generator_marginal_cost)} +``` + +=== "As composed" + + #### Sets + + | Symbol | Meaning | + |---|---| + | $`\mathcal{T}`$ | index $`t`$ — `snapshot` — dispatch periods | + | $`\mathcal{N}`$ | index $`n`$ — `bus` with $`\mathrm{Port\_bus}: \mathcal{J} \to \mathcal{N}`$ — network nodes | + | $`\mathcal{J}`$ | index $`j`$ — `port` with $`\mathrm{Port\_bus}: \mathcal{J} \to \mathcal{N},\ \mathrm{Generator\_port}: \mathcal{G} \to \mathcal{J},\ \mathrm{Load\_port}: \mathcal{D} \to \mathcal{J}`$ — the connections components make, one label per connection | + | $`\mathcal{G}`$ | index $`g`$ — `generator` with $`\mathrm{Generator\_port}: \mathcal{G} \to \mathcal{J}`$ — generating units, each on one port | + | $`\mathcal{D}`$ | index $`d`$ — `load` with $`\mathrm{Load\_port}: \mathcal{D} \to \mathcal{J}`$ — demands, each on one port | + + #### Parameters + + | Symbol | Meaning | + |---|---| + | $`\mathrm{p}^{\mathrm{nom}}`$ | `Generator_p_nom` over $`\mathcal{G}`$ — nominal power | + | $`\mathrm{c}`$ | `Generator_marginal_cost` over $`\mathcal{G}`$ — cost of one unit of output | + | $`\mathrm{load}`$ | `Load_p_set` over $`\mathcal{T} \times \mathcal{D}`$ — `Load-p_set` — what a load takes in a snapshot | + + #### Variables + + | Symbol | Meaning | + |---|---| + | $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — what a port puts into its bus in a snapshot, negative for a withdrawal | + | $`p`$ | `Generator_p` over $`\mathcal{T} \times \mathcal{G}`$ — `Generator-p` — what a generator produces in a snapshot | + + #### Objective + + ```math + \min \sum_{t \in \mathcal{T},\ g \in \mathcal{G}} p_{t,g} \cdot \mathrm{c}_{g} + ``` + + #### Subject to + + **`Bus_nodal_balance`** + + ```math + \sum_{j \in \mathcal{J} \,:\, \mathrm{Port\_bus}(j) = n} f_{t,j} = 0 \qquad \forall\, t \in \mathcal{T},\ n \in \mathcal{N} + ``` + + **`Generator_injection`** + + ```math + f_{t,\mathrm{Generator\_port}(g)} = p_{t,g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} + ``` + + **`Load_withdrawal`** + + ```math + f_{t,\mathrm{Load\_port}(d)} = -\mathrm{load}_{t,d} \qquad \forall\, t \in \mathcal{T},\ d \in \mathcal{D} + ``` + + #### Variable domains + + **`Port_p`** + + ```math + f_{t,j} \in \mathbb{R} \qquad \forall\, t \in \mathcal{T},\ j \in \mathcal{J} + ``` + + **`Generator_p`** + + ```math + 0 \le p_{t,g} \le \mathrm{p}^{\mathrm{nom}}_{g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} + ``` + +=== "With commitment" + + ```yaml title="variants/commitment.yaml" + parameters: + Generator_p_min_pu: { dims: [generator], description: "least output, per unit of nominal power" } + variables: + Generator_status: + dims: [snapshot, generator] + domain: binary + description: "`Generator-status` — whether a unit is on in a snapshot" + Generator_p: { bounds: { upper: .inf } } + constraints: + Generator_com_p_upper: + description: "`Generator-com-p-upper` — a committed unit outputs at most its nominal power; off, at most nothing" + dims: [snapshot, generator] + expression: Generator_p <= Generator_p_nom * Generator_status + Generator_com_p_lower: + description: "`Generator-com-p-lower` — a committed unit outputs at least its minimum; off, at least nothing" + dims: [snapshot, generator] + expression: Generator_p >= Generator_p_min_pu * Generator_p_nom * Generator_status + ``` + + #### Sets + + | Symbol | Meaning | + |---|---| + | $`\mathcal{T}`$ | index $`t`$ — `snapshot` — dispatch periods | + | $`\mathcal{N}`$ | index $`n`$ — `bus` with $`\mathrm{Port\_bus}: \mathcal{J} \to \mathcal{N}`$ — network nodes | + | $`\mathcal{J}`$ | index $`j`$ — `port` with $`\mathrm{Port\_bus}: \mathcal{J} \to \mathcal{N},\ \mathrm{Generator\_port}: \mathcal{G} \to \mathcal{J},\ \mathrm{Load\_port}: \mathcal{D} \to \mathcal{J}`$ — the connections components make, one label per connection | + | $`\mathcal{G}`$ | index $`g`$ — `generator` with $`\mathrm{Generator\_port}: \mathcal{G} \to \mathcal{J}`$ — generating units, each on one port | + | $`\mathcal{D}`$ | index $`d`$ — `load` with $`\mathrm{Load\_port}: \mathcal{D} \to \mathcal{J}`$ — demands, each on one port | + + #### Parameters + + | Symbol | Meaning | + |---|---| + | $`\mathrm{p}^{\mathrm{nom}}`$ | `Generator_p_nom` over $`\mathcal{G}`$ — nominal power | + | $`\mathrm{c}`$ | `Generator_marginal_cost` over $`\mathcal{G}`$ — cost of one unit of output | + | $`\mathrm{load}`$ | `Load_p_set` over $`\mathcal{T} \times \mathcal{D}`$ — `Load-p_set` — what a load takes in a snapshot | + | $`\underline{\mathrm{p}}`$ | `Generator_p_min_pu` over $`\mathcal{G}`$ — least output, per unit of nominal power | + + #### Variables + + | Symbol | Meaning | + |---|---| + | $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — what a port puts into its bus in a snapshot, negative for a withdrawal | + | $`p`$ | `Generator_p` over $`\mathcal{T} \times \mathcal{G}`$ — `Generator-p` — what a generator produces in a snapshot | + | $`u`$ | `Generator_status` over $`\mathcal{T} \times \mathcal{G}`$ — `Generator-status` — whether a unit is on in a snapshot | + + #### Objective + + ```math + \min \sum_{t \in \mathcal{T},\ g \in \mathcal{G}} p_{t,g} \cdot \mathrm{c}_{g} + ``` + + #### Subject to + + **`Bus_nodal_balance`** + + ```math + \sum_{j \in \mathcal{J} \,:\, \mathrm{Port\_bus}(j) = n} f_{t,j} = 0 \qquad \forall\, t \in \mathcal{T},\ n \in \mathcal{N} + ``` + + **`Generator_injection`** + + ```math + f_{t,\mathrm{Generator\_port}(g)} = p_{t,g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} + ``` + + **`Load_withdrawal`** + + ```math + f_{t,\mathrm{Load\_port}(d)} = -\mathrm{load}_{t,d} \qquad \forall\, t \in \mathcal{T},\ d \in \mathcal{D} + ``` + + **`Generator_com_p_upper`** + + ```math + p_{t,g} \le \mathrm{p}^{\mathrm{nom}}_{g} \cdot u_{t,g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} + ``` + + **`Generator_com_p_lower`** + + ```math + p_{t,g} \ge \underline{\mathrm{p}}_{g} \cdot \mathrm{p}^{\mathrm{nom}}_{g} \cdot u_{t,g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} + ``` + + #### Variable domains + + **`Port_p`** + + ```math + f_{t,j} \in \mathbb{R} \qquad \forall\, t \in \mathcal{T},\ j \in \mathcal{J} + ``` + + **`Generator_p`** + + ```math + p_{t,g} \ge 0 \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} + ``` + + **`Generator_status`** + + ```math + u_{t,g} \in \{0, 1\} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} + ``` + diff --git a/docs/examples/library/generator.md b/docs/examples/library/generator.md new file mode 100644 index 00000000..1d6a7e15 --- /dev/null +++ b/docs/examples/library/generator.md @@ -0,0 +1,114 @@ + + +# Generators + +PyPSA's `Generator`, as one fragment. It owns its dimension, its relation into +`port`, its parameters, its column and its cost. It reads `Port_p` from +[the surface](surface.md) under +[`given`](../../reference/language/declarations.md#given). `Generator_port` +stands where PyPSA writes `Generator_bus`. + +The constraint is what makes the library composable. +`at(Port_p, by=Generator_port, over=port, into=generator)` pins the flow at +this component's own port rather than adding a term to the balance, so the +balance does not grow. + +The file is cut to what a dispatch model needs. A fixed build, no availability +profile and no ramp limits are three declarations PyPSA carries and this file +does not. [The PyPSA rungs](../pypsa.md) state them in full. + +The math below is what this file prints on its own, with `Port_p` under +*Given* in the legend. Merged with the surface, `Port_p` is one declaration +again. + + +```yaml +description: >- + PyPSA's `Generator`, wired to a port rather than straight to a bus, and cut + to what a dispatch model needs: a fixed build, no availability profile, no + ramp limits. +dimensions: + snapshot: { dtype: datetime, description: dispatch periods } + port: { dtype: str, description: "the connections components make, one label per connection" } + generator: { dtype: str, description: "generating units, each on one port" } +relations: + Generator_port: { key: generator, values: port } +given: + variables: + Port_p: + dims: [snapshot, port] + description: the surface introduces this column, and this file pins it at its own ports +parameters: + Generator_p_nom: { dims: [generator], description: nominal power } + Generator_marginal_cost: { dims: [generator], description: cost of one unit of output } +variables: + Generator_p: + dims: [snapshot, generator] + bounds: { lower: 0, upper: Generator_p_nom } + description: "`Generator-p` — what a generator produces in a snapshot" +constraints: + Generator_injection: + description: >- + what a generator produces is what its port injects. No PyPSA row stands + for this: PyPSA writes the generator into the balance instead + dims: [snapshot, generator] + expression: at(Port_p, by=Generator_port, over=port, into=generator) == Generator_p +objective: + sense: minimize + expression: sum(Generator_p * Generator_marginal_cost) +``` + +PyPSA's `Generator`, wired to a port rather than straight to a bus, and cut to what a dispatch model needs: a fixed build, no availability profile, no ramp limits. + +#### Sets + +| Symbol | Meaning | +|---|---| +| $`\mathcal{T}`$ | index $`t`$ — `snapshot` — dispatch periods | +| $`\mathcal{J}`$ | index $`j`$ — `port` with $`\mathrm{Generator\_port}: \mathcal{G} \to \mathcal{J}`$ — the connections components make, one label per connection | +| $`\mathcal{G}`$ | index $`g`$ — `generator` with $`\mathrm{Generator\_port}: \mathcal{G} \to \mathcal{J}`$ — generating units, each on one port | + +#### Parameters + +| Symbol | Meaning | +|---|---| +| $`\mathrm{p}^{\mathrm{nom}}`$ | `Generator_p_nom` over $`\mathcal{G}`$ — nominal power | +| $`\mathrm{c}`$ | `Generator_marginal_cost` over $`\mathcal{G}`$ — cost of one unit of output | + +#### Variables + +| Symbol | Meaning | +|---|---| +| $`p`$ | `Generator_p` over $`\mathcal{T} \times \mathcal{G}`$ — `Generator-p` — what a generator produces in a snapshot | + +#### Given + +| Symbol | Meaning | +|---|---| +| $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — the surface introduces this column, and this file pins it at its own ports | + +#### Objective + +```math +\min \sum_{t \in \mathcal{T},\ g \in \mathcal{G}} p_{t,g} \cdot \mathrm{c}_{g} +``` + +#### Subject to + +**`Generator_injection`** + +```math +f_{t,\mathrm{Generator\_port}(g)} = p_{t,g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} +``` + +#### Variable domains + +**`Generator_p`** + +```math +0 \le p_{t,g} \le \mathrm{p}^{\mathrm{nom}}_{g} \qquad \forall\, t \in \mathcal{T},\ g \in \mathcal{G} +``` + diff --git a/docs/examples/library/index.md b/docs/examples/library/index.md new file mode 100644 index 00000000..6e5233e7 --- /dev/null +++ b/docs/examples/library/index.md @@ -0,0 +1,66 @@ + + +# A component library + +Several files that each say part of a model, and compose into one. The surface +declares what the components share. Each component file declares its own math +against the surface, and [`merge`](../../howto/compose.md) makes the model. +Every file here loads and prints on its own, so the unit you pick from is the +unit you can read. + +The names are PyPSA's, spelled `Component_attribute` as +[the PyPSA rungs](../pypsa.md) spell them, and the math prints in the symbols +those pages use. The model is cut to dispatch: one build, no availability +profile, no ramp limits. + +## The layout + +```text +examples/library/ + surface.yaml one flow per port, one balance per bus + generator.yaml PyPSA's Generator + load.yaml PyPSA's Load + variants/ + commitment.yaml a patch over the composition, not a peer +``` + +| Page | What it shows | +| ---------------------------------- | -------------------------------------------------------------- | +| [The coupling surface](surface.md) | the spine, and the sign convention | +| [Generators](generator.md) | a file that reads `Port_p` and prices its output | +| [Loads](load.md) | a file with no variable of its own | +| [The composed model](composed.md) | what `merge` returns, and the math it prints with each variant | + +## Rules of the layout + +- **One file per thing you would pick on its own.** `merge` takes a whole + fragment or none of it, so a model with no storage never mentions storage. +- **One surface.** Two surface files would be two conventions, and no + component file could say which one it meant. +- **Every name carries the component class it belongs to.** `merge` does not + rename, so `Generator_` and `Load_` keep the files apart. The surface owns + `Port_`, `port` and `bus`. +- **A fragment is what a system has. A patch is how a component is + formulated.** A second kind of component is a peer, and `merge` composes it. + A different formulation of one component edits declarations that already + exist, and `override` lays it over the composition. + +## The variant + +`variants/commitment.yaml` makes the generator a committed unit. It adds a +binary, lifts the upper bound the capacity gave `Generator_p`, and caps output +with a constraint instead. + +It edits `Generator_p`, which `generator.yaml` introduces, and names +`Generator_p_nom`, which `generator.yaml` declares. So it is not a model, and +`to_spec` refuses it on its own. It is laid over the composition: + +```python +ms.override(ms.merge(fragments), {'commitment': 'variants/commitment.yaml'}) +``` + +The [composed model](composed.md) carries the patch and the math it makes, in a +tab of its own. That tab is the only place the patch can be read as math. diff --git a/docs/examples/library/load.md b/docs/examples/library/load.md new file mode 100644 index 00000000..1fa76c3c --- /dev/null +++ b/docs/examples/library/load.md @@ -0,0 +1,69 @@ + + +# Loads + +PyPSA's `Load`, and the fragment that shows what a file may leave out. It +declares no variable and no objective. `Load_p_set` is data, and the only thing +the file says is what the load's port withdraws. + +The minus sign is the whole of its relation to the sign convention. A +withdrawal is a negative injection. + + +```yaml +description: PyPSA's `Load`, wired to a port rather than straight to a bus. What it takes is data, so it decides nothing. +dimensions: + snapshot: { dtype: datetime, description: dispatch periods } + port: { dtype: str, description: "the connections components make, one label per connection" } + load: { dtype: str, description: "demands, each on one port" } +relations: + Load_port: { key: load, values: port } +given: + variables: + Port_p: + dims: [snapshot, port] + description: the surface introduces this column, and this file pins it at its own ports +parameters: + Load_p_set: { dims: [snapshot, load], description: "`Load-p_set` — what a load takes in a snapshot" } +constraints: + Load_withdrawal: + description: >- + what a load takes is what its port withdraws. No PyPSA row stands for + this: PyPSA writes the load into the balance instead + dims: [snapshot, load] + expression: at(Port_p, by=Load_port, over=port, into=load) == -Load_p_set +``` + +PyPSA's `Load`, wired to a port rather than straight to a bus. What it takes is data, so it decides nothing. + +#### Sets + +| Symbol | Meaning | +|---|---| +| $`\mathcal{T}`$ | index $`t`$ — `snapshot` — dispatch periods | +| $`\mathcal{J}`$ | index $`j`$ — `port` with $`\mathrm{Load\_port}: \mathcal{D} \to \mathcal{J}`$ — the connections components make, one label per connection | +| $`\mathcal{D}`$ | index $`d`$ — `load` with $`\mathrm{Load\_port}: \mathcal{D} \to \mathcal{J}`$ — demands, each on one port | + +#### Parameters + +| Symbol | Meaning | +|---|---| +| $`\mathrm{load}`$ | `Load_p_set` over $`\mathcal{T} \times \mathcal{D}`$ — `Load-p_set` — what a load takes in a snapshot | + +#### Given + +| Symbol | Meaning | +|---|---| +| $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — the surface introduces this column, and this file pins it at its own ports | + +#### Subject to + +**`Load_withdrawal`** + +```math +f_{t,\mathrm{Load\_port}(d)} = -\mathrm{load}_{t,d} \qquad \forall\, t \in \mathcal{T},\ d \in \mathcal{D} +``` + diff --git a/docs/examples/library/surface.md b/docs/examples/library/surface.md new file mode 100644 index 00000000..3170cbc0 --- /dev/null +++ b/docs/examples/library/surface.md @@ -0,0 +1,77 @@ + + +# The coupling surface + +The spine every other file in the library is written against. It declares one +`Port_p` per port, one balance per bus, and the relation that says which bus a +port sits on. Nothing in it names a component class, so it is the one file that +does not change when a component class is added. + +PyPSA gives each component class a bus column and sums the classes into +`Bus-nodal_balance`. Here a component is wired to a port and the port to a bus, +so the balance sums ports and stays as written however many fragments merge. +The [how-to guide](../../howto/compose.md) shows the same shape with fewer +names. + +A flow is positive where the port injects into its bus. Every component reads +that convention, and no component restates it. + + +```yaml +description: >- + The coupling surface every component in this library is written against: one + flow per port, and one balance per bus. A component is wired to a port, the + port to a bus, and the balance names no component class. A flow is positive + where the port injects into its bus. +dimensions: + snapshot: { dtype: datetime, description: dispatch periods } + bus: { dtype: str, description: network nodes } + port: { dtype: str, description: "the connections components make, one label per connection" } +relations: + Port_bus: { key: port, values: bus } +variables: + Port_p: + dims: [snapshot, port] + description: what a port puts into its bus in a snapshot, negative for a withdrawal +constraints: + Bus_nodal_balance: + description: "`Bus-nodal_balance` — what the ports on a bus put in nets to nothing" + dims: [snapshot, bus] + expression: sum(Port_p, by=Port_bus, over=port, into=bus) == 0 +``` + +The coupling surface every component in this library is written against: one flow per port, and one balance per bus. A component is wired to a port, the port to a bus, and the balance names no component class. A flow is positive where the port injects into its bus. + +#### Sets + +| Symbol | Meaning | +|---|---| +| $`\mathcal{T}`$ | index $`t`$ — `snapshot` — dispatch periods | +| $`\mathcal{N}`$ | index $`n`$ — `bus` with $`\mathrm{Port\_bus}: \mathcal{J} \to \mathcal{N}`$ — network nodes | +| $`\mathcal{J}`$ | index $`j`$ — `port` with $`\mathrm{Port\_bus}: \mathcal{J} \to \mathcal{N}`$ — the connections components make, one label per connection | + +#### Variables + +| Symbol | Meaning | +|---|---| +| $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — what a port puts into its bus in a snapshot, negative for a withdrawal | + +#### Subject to + +**`Bus_nodal_balance`** + +```math +\sum_{j \in \mathcal{J} \,:\, \mathrm{Port\_bus}(j) = n} f_{t,j} = 0 \qquad \forall\, t \in \mathcal{T},\ n \in \mathcal{N} +``` + +#### Variable domains + +**`Port_p`** + +```math +f_{t,j} \in \mathbb{R} \qquad \forall\, t \in \mathcal{T},\ j \in \mathcal{J} +``` + diff --git a/examples/library/generator.yaml b/examples/library/generator.yaml new file mode 100644 index 00000000..ffb5c604 --- /dev/null +++ b/examples/library/generator.yaml @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +description: >- + PyPSA's `Generator`, wired to a port rather than straight to a bus, and cut + to what a dispatch model needs: a fixed build, no availability profile, no + ramp limits. +dimensions: + snapshot: { dtype: datetime, description: dispatch periods } + port: { dtype: str, description: "the connections components make, one label per connection" } + generator: { dtype: str, description: "generating units, each on one port" } +relations: + Generator_port: { key: generator, values: port } +given: + variables: + Port_p: + dims: [snapshot, port] + description: the surface introduces this column, and this file pins it at its own ports +parameters: + Generator_p_nom: { dims: [generator], description: nominal power } + Generator_marginal_cost: { dims: [generator], description: cost of one unit of output } +variables: + Generator_p: + dims: [snapshot, generator] + bounds: { lower: 0, upper: Generator_p_nom } + description: "`Generator-p` — what a generator produces in a snapshot" +constraints: + Generator_injection: + description: >- + what a generator produces is what its port injects. No PyPSA row stands + for this: PyPSA writes the generator into the balance instead + dims: [snapshot, generator] + expression: at(Port_p, by=Generator_port, over=port, into=generator) == Generator_p +objective: + sense: minimize + expression: sum(Generator_p * Generator_marginal_cost) diff --git a/examples/library/load.yaml b/examples/library/load.yaml new file mode 100644 index 00000000..dbbfe2b9 --- /dev/null +++ b/examples/library/load.yaml @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +description: PyPSA's `Load`, wired to a port rather than straight to a bus. What it takes is data, so it decides nothing. +dimensions: + snapshot: { dtype: datetime, description: dispatch periods } + port: { dtype: str, description: "the connections components make, one label per connection" } + load: { dtype: str, description: "demands, each on one port" } +relations: + Load_port: { key: load, values: port } +given: + variables: + Port_p: + dims: [snapshot, port] + description: the surface introduces this column, and this file pins it at its own ports +parameters: + Load_p_set: { dims: [snapshot, load], description: "`Load-p_set` — what a load takes in a snapshot" } +constraints: + Load_withdrawal: + description: >- + what a load takes is what its port withdraws. No PyPSA row stands for + this: PyPSA writes the load into the balance instead + dims: [snapshot, load] + expression: at(Port_p, by=Load_port, over=port, into=load) == -Load_p_set diff --git a/examples/library/surface.yaml b/examples/library/surface.yaml new file mode 100644 index 00000000..199c4d52 --- /dev/null +++ b/examples/library/surface.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +description: >- + The coupling surface every component in this library is written against: one + flow per port, and one balance per bus. A component is wired to a port, the + port to a bus, and the balance names no component class. A flow is positive + where the port injects into its bus. +dimensions: + snapshot: { dtype: datetime, description: dispatch periods } + bus: { dtype: str, description: network nodes } + port: { dtype: str, description: "the connections components make, one label per connection" } +relations: + Port_bus: { key: port, values: bus } +variables: + Port_p: + dims: [snapshot, port] + description: what a port puts into its bus in a snapshot, negative for a withdrawal +constraints: + Bus_nodal_balance: + description: "`Bus-nodal_balance` — what the ports on a bus put in nets to nothing" + dims: [snapshot, bus] + expression: sum(Port_p, by=Port_bus, over=port, into=bus) == 0 diff --git a/examples/library/variants/commitment.yaml b/examples/library/variants/commitment.yaml new file mode 100644 index 00000000..546e1bec --- /dev/null +++ b/examples/library/variants/commitment.yaml @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +parameters: + Generator_p_min_pu: { dims: [generator], description: "least output, per unit of nominal power" } +variables: + Generator_status: + dims: [snapshot, generator] + domain: binary + description: "`Generator-status` — whether a unit is on in a snapshot" + Generator_p: { bounds: { upper: .inf } } +constraints: + Generator_com_p_upper: + description: "`Generator-com-p-upper` — a committed unit outputs at most its nominal power; off, at most nothing" + dims: [snapshot, generator] + expression: Generator_p <= Generator_p_nom * Generator_status + Generator_com_p_lower: + description: "`Generator-com-p-lower` — a committed unit outputs at least its minimum; off, at least nothing" + dims: [snapshot, generator] + expression: Generator_p >= Generator_p_min_pu * Generator_p_nom * Generator_status diff --git a/examples/symbols/library.yaml b/examples/symbols/library.yaml new file mode 100644 index 00000000..f251797c --- /dev/null +++ b/examples/symbols/library.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +# How `examples/library/` prints. The names are PyPSA's, so a fragment reads +# beside `n.model`; these symbols are the ones `pypsa.yaml` uses. One table +# serves every fragment, the model they compose and the variants laid over it. +# Each page takes the cut of it that its own model declares, because a table +# naming anything else is refused. +notation: latex +dimensions: + snapshot: { index: t, set: '\mathcal{T}' } + bus: { index: n, set: '\mathcal{N}' } + port: { index: j, set: '\mathcal{J}' } + generator: { index: g, set: '\mathcal{G}' } + load: { index: d, set: '\mathcal{D}' } +names: + Port_p: f + Generator_p: p + Generator_p_nom: '\mathrm{p}^{\mathrm{nom}}' + Generator_marginal_cost: '\mathrm{c}' + Generator_p_min_pu: '\underline{\mathrm{p}}' + Generator_status: u + Load_p_set: '\mathrm{load}' diff --git a/mkdocs.yml b/mkdocs.yml index e7107826..241676a5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -69,6 +69,12 @@ nav: - PyPSA, the lossy lines: examples/pypsa_losses.md - PyPSA, the two-stage class: examples/pypsa_stochastic.md - PyPSA, the multi-period class: examples/pypsa_multi_period.md + - A component library: + - examples/library/index.md + - The coupling surface: examples/library/surface.md + - Generators: examples/library/generator.md + - Loads: examples/library/load.md + - The composed model: examples/library/composed.md # Everything a reader does not need in order to write a model: the design # arguments, how to contribute, what changed. - About: diff --git a/tests/test_library_example.py b/tests/test_library_example.py new file mode 100644 index 00000000..1d5ddf85 --- /dev/null +++ b/tests/test_library_example.py @@ -0,0 +1,102 @@ +# SPDX-FileCopyrightText: math-spec Contributors +# +# SPDX-License-Identifier: MIT + +"""The component library under `examples/library/`, held to what its pages claim. + +The gallery test holds each page to its generator, so the math on a page cannot +drift from the file above it. What is left for here is what no page states: that +every fragment stands alone, that the composition is one model, and that the +variant patch, the one file in the library that is not a model, applies to what +the fragments make. +""" + +from __future__ import annotations + +import pytest + +from math_spec import LanguageError, merge, override, to_markdown, to_spec +from math_spec.typesetting import FORMATS, typeset +from tests.fixtures import EXAMPLES +from tools import gallery + +LIBRARY = EXAMPLES / 'library' +FRAGMENTS = {name: LIBRARY / f'{name}.yaml' for name in ('surface', 'generator', 'load')} +PATCH = LIBRARY / 'variants' / 'commitment.yaml' +PATCHED = override(merge(FRAGMENTS), {'commitment': PATCH}) + + +@pytest.mark.parametrize('name', sorted(FRAGMENTS)) +def test_every_fragment_loads_and_prints_on_its_own(name): + """The unit a library ships is the unit somebody reviews, so each one is a model.""" + assert to_markdown(to_spec(FRAGMENTS[name])), f'{name} rendered nothing' + + +@pytest.mark.parametrize('name', ['generator', 'load']) +def test_a_component_file_reads_the_surface_and_introduces_no_flow(name): + spec = to_spec(FRAGMENTS[name]) + assert sorted(spec.given.variables) == ['Port_p'], 'the flow is the one column a component file reads' + assert 'Port_p' not in spec.variables, 'the surface introduces the column, and a component file only reads it' + + +def test_the_library_composes_into_one_model(): + spec = to_spec(merge(FRAGMENTS)) + assert sorted(spec.variables) == ['Generator_p', 'Port_p'], 'the composition declares each column once' + assert sorted(spec.constraints) == ['Bus_nodal_balance', 'Generator_injection', 'Load_withdrawal'], ( + 'the composition carries every row family of every fragment, and no other' + ) + assert not spec.given, 'each read is folded into the declaration that introduces it' + assert spec.objective is not None and spec.objective.expression == 'sum(Generator_p * Generator_marginal_cost)', ( + "the one fragment that priced anything carries the composed model's objective, as it wrote it" + ) + + +def test_the_balance_is_written_once_however_many_fragments_are_merged(): + one = to_spec(merge({'surface': FRAGMENTS['surface'], 'load': FRAGMENTS['load']})) + both = to_spec(merge(FRAGMENTS)) + assert one.constraints['Bus_nodal_balance'] == both.constraints['Bus_nodal_balance'], ( + 'a component file pins the flow at its own port, so adding one leaves the balance as the surface wrote it' + ) + + +def test_the_variant_is_a_patch_rather_than_a_model(): + """Which is why `render_tex` skips `variants/`: nothing there loads on its own.""" + with pytest.raises(LanguageError): + to_spec(PATCH) + + +def test_the_variant_patch_applies_to_the_composition(): + spec = to_spec(PATCHED) + assert spec.variables['Generator_status'].domain == 'binary' + assert spec.variables['Generator_p'].bounds.upper == float('inf'), 'the cap moves from the bound to a constraint' + assert sorted(spec.constraints) == [ + 'Bus_nodal_balance', + 'Generator_com_p_lower', + 'Generator_com_p_upper', + 'Generator_injection', + 'Load_withdrawal', + ], 'the patch adds its two rows and removes none' + + +@pytest.mark.parametrize('fmt', list(FORMATS), ids=list(FORMATS)) +def test_the_patched_model_prints_the_variant_math(fmt): + """A patch is read by the loader through the model it lands on, so what it declares prints like the rest.""" + printed = typeset(to_spec(PATCHED), fmt).replace(r'\_', '_') + declared = ('Generator_status', 'Generator_p_min_pu', 'Generator_com_p_upper', 'Generator_com_p_lower') + missing = [name for name in declared if name not in printed] + assert not missing, f'the patch declares {missing}, and the typeset document does not name them' + + +def test_the_variant_needs_the_fragment_it_patches(): + """Picking commitment without the generator is a patch that lands on nothing, and it is refused at load.""" + without_generator = merge({name: FRAGMENTS[name] for name in ('surface', 'load')}) + with pytest.raises(LanguageError, match="edits the variable 'Generator_p', which its base does not declare"): + to_spec(override(without_generator, {'commitment': PATCH})) + + +def test_every_variant_in_the_library_is_typeset_on_the_composed_page(): + """A patch prints only as the model it lands on, so one with no tab is a patch nothing prints.""" + _, patches = gallery.COMPOSED['library/composed.md'] + assert patches == {path.stem: path for path in (LIBRARY / 'variants').glob('*.yaml')}, ( + 'every file under variants/ takes a tab on the composed page, under its own name' + ) diff --git a/tools/_page.py b/tools/_page.py index 35a2b264..9048d7f8 100644 --- a/tools/_page.py +++ b/tools/_page.py @@ -9,6 +9,7 @@ import argparse import re import sys +import textwrap from pathlib import Path from typing import TYPE_CHECKING @@ -44,6 +45,11 @@ def inlined(markdown: str) -> str: return FENCE.sub(lambda m: f'$`{" ".join(m[1].splitlines())}`$', markdown) +def tab(title: str, body: str) -> str: + """One tab of a tabbed block: its title, and its body indented into it.""" + return f'=== "{title}"\n\n{textwrap.indent(body, " ")}' + + def without_header(path: Path) -> str: """The file from its first line that is neither blank nor a comment: the licence header is the repository's.""" lines = path.read_text().splitlines() diff --git a/tools/gallery.py b/tools/gallery.py index 14a4a1ec..b05f6678 100644 --- a/tools/gallery.py +++ b/tools/gallery.py @@ -17,11 +17,13 @@ import re import textwrap from functools import partial -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any -from math_spec import to_spec +import yaml + +from math_spec import merge, override, to_spec from math_spec.typesetting import to_markdown -from tools._page import ROOT, sidecar_for, splice, without_header +from tools._page import ROOT, sidecar_for, splice, tab, without_header from tools._page import main as page_main from tools.notation import equations from tools.spec_math import OPERATORS, PROBES, _section, rendered_probe @@ -29,7 +31,14 @@ if TYPE_CHECKING: from pathlib import Path + from math_spec.model import Spec + PAGES = ROOT / 'docs' / 'examples' +LIBRARY = ROOT / 'examples' / 'library' +#: How `examples/library/` prints: one table for every fragment, the model +#: they compose and each variant laid over it. `symbols_for` cuts it to what +#: one model declares, because a table naming anything else is refused. +LIBRARY_SYMBOLS = ROOT / 'examples' / 'symbols' / 'library.yaml' BEGIN, END = '', '' #: Page -> the model it shows. One model per page, because a gallery of @@ -37,6 +46,20 @@ MODELS = { 'dispatch.md': ROOT / 'examples' / 'dispatch.yaml', 'commitment.md': ROOT / 'examples' / 'commitment.yaml', + 'library/surface.md': LIBRARY / 'surface.yaml', + 'library/generator.md': LIBRARY / 'generator.yaml', + 'library/load.md': LIBRARY / 'load.yaml', +} + +#: Page -> the fragments whose composition it shows, and the patches laid over +#: it. The model is what `merge` returns, which no file in the tree holds, so +#: the page carries it as YAML beside the math it prints. A patch has no math +#: of its own, so each one prints as the model it lands on, in a tab of its own. +COMPOSED = { + 'library/composed.md': ( + [LIBRARY / name for name in ('surface.yaml', 'generator.yaml', 'load.yaml')], + {path.stem: path for path in sorted((LIBRARY / 'variants').glob('*.yaml'))}, + ), } #: Page -> the model it shows one declaration at a time — its YAML, then the @@ -63,6 +86,48 @@ def model_block(path: Path) -> str: return f'```yaml\n{without_header(path)}\n```\n\n{to_markdown(path, numbered=False).strip()}' +def symbols_for(model: Spec) -> dict[str, Any]: + """The library's symbol table, cut to the dimensions and names *model* declares.""" + table = yaml.safe_load(LIBRARY_SYMBOLS.read_text()) + named = {*model.parameters, *model.variables, *model.given.variables, *model.expressions, *model.constraints} + return { + 'notation': table['notation'], + 'dimensions': {name: symbol for name, symbol in table['dimensions'].items() if name in model.dimensions}, + 'names': {name: symbol for name, symbol in table['names'].items() if name in named}, + } + + +def library_block(path: Path) -> str: + """One fragment of the library, then its document in the notation the whole library prints in.""" + model = to_spec(path) + printed = to_markdown(model, symbols=symbols_for(model), numbered=False) + return f'```yaml\n{without_header(path)}\n```\n\n{printed.strip()}' + + +def composed_block(fragments: list[Path], patches: dict[str, Path]) -> str: + """The mapping `merge` returns for *fragments* as YAML, then its document as composed and under each patch. + + The composed YAML is generated rather than committed, so the page cannot + show a composition the fragments beside it no longer make. A patch is + refused on its own, so its tab carries the patch file and then the whole + document of the model it is laid over. + """ + composed = merge({path.stem: path for path in fragments}) + model = to_spec(composed) + tabs = [tab('As composed', to_markdown(model, symbols=symbols_for(model), numbered=False).strip())] + for name, path in patches.items(): + patched = to_spec(override(composed, {name: path})) + tabs.append( + tab( + f'With {name}', + f'```yaml title="variants/{path.name}"\n{without_header(path)}\n```\n\n' + f'{to_markdown(patched, symbols=symbols_for(patched), numbered=False).strip()}', + ) + ) + dumped = yaml.safe_dump(composed, sort_keys=False, default_flow_style=None, allow_unicode=True, width=100).strip() + return f'```yaml\n{dumped}\n```\n\n' + '\n\n'.join(tabs) + + def probe_block() -> str: """Every operator probe: the model, then the one equation it renders.""" parts = [] @@ -183,6 +248,10 @@ def block(page: str) -> str: return probe_block() if page in DECLARED: return declared_block(DECLARED[page]) + if page in COMPOSED: + return composed_block(*COMPOSED[page]) + if page.startswith('library/'): + return library_block(MODELS[page]) return model_block(MODELS[page]) @@ -194,7 +263,7 @@ def rendered(page: str, text: str) -> str: def pages() -> list[str]: - return [*MODELS, *DECLARED, 'operators.md'] + return [*MODELS, *COMPOSED, *DECLARED, 'operators.md'] def main(argv: list[str] | None = None) -> int: diff --git a/tools/home_math.py b/tools/home_math.py index a94d54ec..b9818928 100644 --- a/tools/home_math.py +++ b/tools/home_math.py @@ -16,11 +16,9 @@ from __future__ import annotations -import textwrap - from math_spec import to_spec from math_spec.typesetting import to_latex, to_markdown, to_typst -from tools._page import ROOT, inlined, sidecar_for, splice, without_header +from tools._page import ROOT, inlined, sidecar_for, splice, tab, without_header from tools._page import main as page_main PAGE = ROOT / 'docs' / 'index.md' @@ -76,11 +74,6 @@ loads.""" -def tab(title: str, body: str) -> str: - """One tab of the block: its title, and its body indented into it.""" - return f'=== "{title}"\n\n{textwrap.indent(body, " ")}' - - def block() -> str: """The three tabs, in the order a reader meets them.""" spec = to_spec(MODEL) diff --git a/tools/render_tex.py b/tools/render_tex.py index 8a53a9e2..b8a858c1 100644 --- a/tools/render_tex.py +++ b/tools/render_tex.py @@ -22,8 +22,10 @@ #: Every model the repository has; `examples/*.yaml` is not recursive, and a glob that narrows is a gate that stops testing. CORPUS = ('examples/**/*.yaml', 'tests/typesetting/golden/*.yaml') -#: Inside that glob and not models: the symbol tables `sidecar_for` looks up. -NOT_MODELS = ('examples/symbols',) +#: Inside that glob and not models: the symbol tables `sidecar_for` looks up, +#: and the patches a library's variants are written as, which `override` lays +#: over a model rather than anything loading them on their own. +NOT_MODELS = ('examples/symbols', 'examples/library/variants') def models() -> list[Path]: From 16777b6b206ab06c73576bdab989ff31010a2f2d Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 19:32:06 +0200 Subject: [PATCH 8/9] docs(examples): the library pages state a rule once, and name the port flow one way The layout rules keep the rule sentence and drop the clause that argued for it. The library pages say "surface" where one said "spine", and the component fragments call `Port_p` a flow, as the surface and its page already do. The composed page's variant test reads the page and asks for a tab per file under `variants/`, rather than comparing a glob with itself. The balance test compares each merge against the surface's own row, which is what the page claims. The gallery picks the library block by the model's directory. --- docs/examples/library/generator.md | 4 ++-- docs/examples/library/index.md | 7 +++---- docs/examples/library/load.md | 4 ++-- docs/examples/library/surface.md | 2 +- examples/library/generator.yaml | 2 +- examples/library/load.yaml | 2 +- tests/test_library_example.py | 28 +++++++++++++++++----------- tools/gallery.py | 2 +- 8 files changed, 28 insertions(+), 23 deletions(-) diff --git a/docs/examples/library/generator.md b/docs/examples/library/generator.md index 1d6a7e15..f28c1c6d 100644 --- a/docs/examples/library/generator.md +++ b/docs/examples/library/generator.md @@ -40,7 +40,7 @@ given: variables: Port_p: dims: [snapshot, port] - description: the surface introduces this column, and this file pins it at its own ports + description: the surface introduces this flow, and this file pins it at its own ports parameters: Generator_p_nom: { dims: [generator], description: nominal power } Generator_marginal_cost: { dims: [generator], description: cost of one unit of output } @@ -88,7 +88,7 @@ PyPSA's `Generator`, wired to a port rather than straight to a bus, and cut to w | Symbol | Meaning | |---|---| -| $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — the surface introduces this column, and this file pins it at its own ports | +| $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — the surface introduces this flow, and this file pins it at its own ports | #### Objective diff --git a/docs/examples/library/index.md b/docs/examples/library/index.md index 6e5233e7..22598739 100644 --- a/docs/examples/library/index.md +++ b/docs/examples/library/index.md @@ -29,7 +29,7 @@ examples/library/ | Page | What it shows | | ---------------------------------- | -------------------------------------------------------------- | -| [The coupling surface](surface.md) | the spine, and the sign convention | +| [The coupling surface](surface.md) | the surface, and the sign convention | | [Generators](generator.md) | a file that reads `Port_p` and prices its output | | [Loads](load.md) | a file with no variable of its own | | [The composed model](composed.md) | what `merge` returns, and the math it prints with each variant | @@ -38,10 +38,9 @@ examples/library/ - **One file per thing you would pick on its own.** `merge` takes a whole fragment or none of it, so a model with no storage never mentions storage. -- **One surface.** Two surface files would be two conventions, and no - component file could say which one it meant. +- **One surface.** Every component file is written against it. - **Every name carries the component class it belongs to.** `merge` does not - rename, so `Generator_` and `Load_` keep the files apart. The surface owns + rename. `Generator_` and `Load_` keep the files apart, and the surface owns `Port_`, `port` and `bus`. - **A fragment is what a system has. A patch is how a component is formulated.** A second kind of component is a peer, and `merge` composes it. diff --git a/docs/examples/library/load.md b/docs/examples/library/load.md index 1fa76c3c..f4fc24fa 100644 --- a/docs/examples/library/load.md +++ b/docs/examples/library/load.md @@ -25,7 +25,7 @@ given: variables: Port_p: dims: [snapshot, port] - description: the surface introduces this column, and this file pins it at its own ports + description: the surface introduces this flow, and this file pins it at its own ports parameters: Load_p_set: { dims: [snapshot, load], description: "`Load-p_set` — what a load takes in a snapshot" } constraints: @@ -57,7 +57,7 @@ PyPSA's `Load`, wired to a port rather than straight to a bus. What it takes is | Symbol | Meaning | |---|---| -| $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — the surface introduces this column, and this file pins it at its own ports | +| $`f`$ | `Port_p` over $`\mathcal{T} \times \mathcal{J}`$ — the surface introduces this flow, and this file pins it at its own ports | #### Subject to diff --git a/docs/examples/library/surface.md b/docs/examples/library/surface.md index 3170cbc0..f0525c91 100644 --- a/docs/examples/library/surface.md +++ b/docs/examples/library/surface.md @@ -5,7 +5,7 @@ SPDX-License-Identifier: CC-BY-4.0 # The coupling surface -The spine every other file in the library is written against. It declares one +The surface every other file in the library is written against. It declares one `Port_p` per port, one balance per bus, and the relation that says which bus a port sits on. Nothing in it names a component class, so it is the one file that does not change when a component class is added. diff --git a/examples/library/generator.yaml b/examples/library/generator.yaml index ffb5c604..fac446f3 100644 --- a/examples/library/generator.yaml +++ b/examples/library/generator.yaml @@ -16,7 +16,7 @@ given: variables: Port_p: dims: [snapshot, port] - description: the surface introduces this column, and this file pins it at its own ports + description: the surface introduces this flow, and this file pins it at its own ports parameters: Generator_p_nom: { dims: [generator], description: nominal power } Generator_marginal_cost: { dims: [generator], description: cost of one unit of output } diff --git a/examples/library/load.yaml b/examples/library/load.yaml index dbbfe2b9..afec066a 100644 --- a/examples/library/load.yaml +++ b/examples/library/load.yaml @@ -13,7 +13,7 @@ given: variables: Port_p: dims: [snapshot, port] - description: the surface introduces this column, and this file pins it at its own ports + description: the surface introduces this flow, and this file pins it at its own ports parameters: Load_p_set: { dims: [snapshot, load], description: "`Load-p_set` — what a load takes in a snapshot" } constraints: diff --git a/tests/test_library_example.py b/tests/test_library_example.py index 1d5ddf85..3af4d8df 100644 --- a/tests/test_library_example.py +++ b/tests/test_library_example.py @@ -35,8 +35,8 @@ def test_every_fragment_loads_and_prints_on_its_own(name): @pytest.mark.parametrize('name', ['generator', 'load']) def test_a_component_file_reads_the_surface_and_introduces_no_flow(name): spec = to_spec(FRAGMENTS[name]) - assert sorted(spec.given.variables) == ['Port_p'], 'the flow is the one column a component file reads' - assert 'Port_p' not in spec.variables, 'the surface introduces the column, and a component file only reads it' + assert sorted(spec.given.variables) == ['Port_p'], 'the port flow is the one name a component file reads' + assert 'Port_p' not in spec.variables, 'the surface introduces the flow, and a component file only reads it' def test_the_library_composes_into_one_model(): @@ -51,11 +51,18 @@ def test_the_library_composes_into_one_model(): ) -def test_the_balance_is_written_once_however_many_fragments_are_merged(): - one = to_spec(merge({'surface': FRAGMENTS['surface'], 'load': FRAGMENTS['load']})) - both = to_spec(merge(FRAGMENTS)) - assert one.constraints['Bus_nodal_balance'] == both.constraints['Bus_nodal_balance'], ( - 'a component file pins the flow at its own port, so adding one leaves the balance as the surface wrote it' +@pytest.mark.parametrize( + 'names', + [ + pytest.param(('surface', 'load'), id='one component file'), + pytest.param(('surface', 'generator', 'load'), id='the whole library'), + ], +) +def test_the_balance_is_written_once_however_many_fragments_are_merged(names): + merged = to_spec(merge({name: FRAGMENTS[name] for name in names})) + surface = to_spec(FRAGMENTS['surface']) + assert merged.constraints['Bus_nodal_balance'] == surface.constraints['Bus_nodal_balance'], ( + 'a component file pins the flow at its own port, so merging leaves the balance as the surface wrote it' ) @@ -96,7 +103,6 @@ def test_the_variant_needs_the_fragment_it_patches(): def test_every_variant_in_the_library_is_typeset_on_the_composed_page(): """A patch prints only as the model it lands on, so one with no tab is a patch nothing prints.""" - _, patches = gallery.COMPOSED['library/composed.md'] - assert patches == {path.stem: path for path in (LIBRARY / 'variants').glob('*.yaml')}, ( - 'every file under variants/ takes a tab on the composed page, under its own name' - ) + page = (gallery.PAGES / 'library' / 'composed.md').read_text() + missing = [path.name for path in (LIBRARY / 'variants').glob('*.yaml') if f'=== "With {path.stem}"' not in page] + assert not missing, f'the composed page gives {missing} no tab, so what they print is on no page' diff --git a/tools/gallery.py b/tools/gallery.py index b05f6678..38c33429 100644 --- a/tools/gallery.py +++ b/tools/gallery.py @@ -250,7 +250,7 @@ def block(page: str) -> str: return declared_block(DECLARED[page]) if page in COMPOSED: return composed_block(*COMPOSED[page]) - if page.startswith('library/'): + if MODELS[page].parent == LIBRARY: return library_block(MODELS[page]) return model_block(MODELS[page]) From 4afb08cd69bee2358600611ec18a70a261a8f063 Mon Sep 17 00:00:00 2001 From: Fabian Date: Sat, 19 Sep 2026 21:21:56 +0200 Subject: [PATCH 9/9] docs: one word per concept and shorter sentences on the given pages --- docs/about/limits.md | 7 ++++--- docs/examples/library/generator.md | 4 ++-- docs/examples/library/index.md | 4 ++-- docs/howto/compose.md | 8 ++++---- docs/reference/language/declarations.md | 4 ++-- docs/reference/language/file.md | 2 +- 6 files changed, 15 insertions(+), 14 deletions(-) diff --git a/docs/about/limits.md b/docs/about/limits.md index 3e049b06..354cd038 100644 --- a/docs/about/limits.md +++ b/docs/about/limits.md @@ -146,7 +146,8 @@ balance. The topology is data. Adding a second battery is a row in a table, so the file grows with the number of component _types_. -A fragment reads the coupling surface it is written against, and declares that +A fragment reads the [coupling surface](../examples/library/surface.md) it is +written against, and declares that column under [`given: variables:`](../reference/language/declarations.md#given). So it loads on its own, and prints as math on its own. @@ -154,8 +155,8 @@ loads on its own, and prints as math on its own. Composition happens before `to_spec`, and two verbs do it. `merge` composes fragments as peers: a name two of them declare is refused, and a given declaration is folded into the fragment that introduces the name. `override` -lays a patch over a base a field at a time, which is what a project that -extends a model it does not own writes instead of a copy. It refuses a patch +lays a patch over a base, one field at a time. A project that extends a model +it does not own writes a patch instead of a copy. It refuses a patch that lands on nothing, two patches that write one field, and a dimension or a relation redeclared under the math. The recipe for both is in [compose a model from several files](../howto/compose.md). diff --git a/docs/examples/library/generator.md b/docs/examples/library/generator.md index f28c1c6d..e2e73269 100644 --- a/docs/examples/library/generator.md +++ b/docs/examples/library/generator.md @@ -21,8 +21,8 @@ profile and no ramp limits are three declarations PyPSA carries and this file does not. [The PyPSA rungs](../pypsa.md) state them in full. The math below is what this file prints on its own, with `Port_p` under -*Given* in the legend. Merged with the surface, `Port_p` is one declaration -again. +*Given* in the legend. When it merges with the surface, `Port_p` is one +declaration again. ```yaml diff --git a/docs/examples/library/index.md b/docs/examples/library/index.md index 22598739..3c00ccc7 100644 --- a/docs/examples/library/index.md +++ b/docs/examples/library/index.md @@ -5,7 +5,7 @@ SPDX-License-Identifier: CC-BY-4.0 # A component library -Several files that each say part of a model, and compose into one. The surface +Several files each say part of a model and compose into one. The surface declares what the components share. Each component file declares its own math against the surface, and [`merge`](../../howto/compose.md) makes the model. Every file here loads and prints on its own, so the unit you pick from is the @@ -38,7 +38,7 @@ examples/library/ - **One file per thing you would pick on its own.** `merge` takes a whole fragment or none of it, so a model with no storage never mentions storage. -- **One surface.** Every component file is written against it. +- **Every component file is written against one surface.** - **Every name carries the component class it belongs to.** `merge` does not rename. `Generator_` and `Load_` keep the files apart, and the surface owns `Port_`, `port` and `bus`. diff --git a/docs/howto/compose.md b/docs/howto/compose.md index e647d2ec..fc9f92aa 100644 --- a/docs/howto/compose.md +++ b/docs/howto/compose.md @@ -15,7 +15,7 @@ file, and the two compose as `override(merge({…}), {…})`. ## A library of components 1. **Write the coupling surface as a model.** One flow per port, one balance - per bus. Nothing in it names a component type. + per bus. Nothing in it names a component class. ```yaml title="surface.yaml" dimensions: @@ -99,7 +99,7 @@ file, and the two compose as `override(merge({…}), {…})`. of the fragments are summed, each term in parentheses, in the order the fragment names sort in. -4. **Add a component type without touching the balance.** A component file +4. **Add a component class without touching the balance.** A component file pins the flow at its own port rather than adding a term to the balance, so `Bus_balance` is written once and stays as it is however many files are merged. What grows is the data: which ports exist, and which bus each one @@ -143,8 +143,8 @@ a difference is refused as it is for a dimension. ## A name one fragment both builds and reads A fragment reads what another file builds. A fragment that declares a name and -reads it as well is a file `to_spec` refuses on its own, so `merge` refuses it -too rather than folding the reading away: +reads it as well is a file `to_spec` refuses on its own. So `merge` refuses it +too, rather than folding the reading away: ```text fragment 'generator' declares the variable 'Generator_p' and reads it under 'given: variables:' as well. A given declaration is what one file expects of another, and this fragment builds the name itself: drop the given entry, or move the declaration to the fragment this one reads it from. diff --git a/docs/reference/language/declarations.md b/docs/reference/language/declarations.md index ad4e1762..04869dab 100644 --- a/docs/reference/language/declarations.md +++ b/docs/reference/language/declarations.md @@ -139,8 +139,8 @@ declaration for a consumer to bind ### `given: constraints` -A given constraint is a row family this file reads the dual of and another -model builds. +A given constraint is a row family that another model builds. This file reads +its dual. ```yaml dimensions: diff --git a/docs/reference/language/file.md b/docs/reference/language/file.md index ec78198f..93e1d658 100644 --- a/docs/reference/language/file.md +++ b/docs/reference/language/file.md @@ -14,7 +14,7 @@ and `description`. Any subset of the eleven is accepted. | `relations` | named relations between dimensions ([relations](relations.md)) | | `parameters` | the data the model expects ([declarations](declarations.md)) | | `variables` | what the solver decides | -| `given` | what this file reads and another file builds ([given](declarations.md#given)) | +| `given` | what this file reads but does not build ([given](declarations.md#given)) | | `constraints` | the rules those decisions obey | | `objective` | what is minimised or maximised | | `expressions` | named quantities, reusable in the math and readable after a solve ([named expressions](named.md)) |