From 484fe8532f9d238c55a034728f87abb0afe6f219 Mon Sep 17 00:00:00 2001 From: Juliette-Gerbaux <130555142+Juliette-Gerbaux@users.noreply.github.com> Date: Mon, 24 Aug 2026 10:32:42 +0200 Subject: [PATCH 1/4] Add lower_bound()/upper_bound() expression operators (#264) * Add first tests on thermal heuristics * Implement accurate heuristic on simple problem * Add comments * Add fast heuristic and improve accurate * rename test heuristic * Implement complex model * Accurate for complex model * Type errors and useless solver parameters * check hourly outputs * fast heuristic for complex case * Test for 2 weeks * Test for 2 scenarios * add test for scenarios * Add more details on tests concerning scenarios * Add milp test for second case * remove milp test * Add test accurate for second test case * Add fast test for second test case * convert fast to optimization problem * Optimization problem for fast on simple case * Optimization model fast complex case * Small corrections * Correct test * Optimization problem heuristic fast second complex test case * Equivalent solution for fast heuristic optimization * Api for fast and accurate models * Api for accurate heuristic model * Move models to library * Move some functions from tests to src * Remove all functions from simple test * Move functions from second complex test to src * Remove functions from second test and filter ts on scenarios and timesteps * Sort imports * Run black * Change solver to fix tests in the ci * Fix ci * Rename tests * Generalize check_output * More tests on data * Refactor model * Refactor problem.py * Generic database * Correct error * Create ThermalProblemBuilder * Resolution steps * Simplify get_value out of database * Simplify update of database * Fix ci * Remove not used imports * New test * Small changes * New test with bc * Refactoring * Exhibit models * Remove unused imports * New class for time scenario parameters * Refactor solve * Sort imports * Formating * Refactor data_path * Fix test * Refactor models * Fix test * Run isort * Week scenario parameters * Refactor solve * Rename functions * Refactor cluster parameters * Correct error * Refactor update * New test * New test with ramp * Refactor * Add description for tests * Beginning of new test * Improve edit_value * Improve cluster parameters * Move expected_output class * Change names * Remove network and database building from thermal_problem_builder * Correct pytest fixture * Remove solve from thermal problem builder * Fix test * Move tests * Improve test with parameters * Use data_path * Improve pytest fixture * Improve tests * Fix ci * Fix tests * Fix test on day ahead reserve * Unused code * api file * Small changes * Move files * Prepare test_one_cluster * Update tests * Update test with bc * Update test with different scenarios and xpress settings * Remove test with different scenarios and xpress settings * Remove day ahead test * Implement fast heuristic as an algorithm * Implement accurate heuristic * integer-strategy in system and heuristic in optim-config * Implement interger-strategy in variables building * heuristic id in parsing * Add heuristic validation and update optim-config documentation * Workflow with 2 iterations and thermal heuristic * Remove obsolete code * Fix ci * Test with ramp * Validate heuristic input/output time-dependence at optim-config parsing Cross-check each heuristic input/output declared in optim-config.yml against the model: the referenced id must exist and have the time-dependence the fast/accurate thermal heuristics expect (e.g. min_up_duration constant, generation_power per-timestep), catching mismatches at load time instead of a runtime crash mid-solve. nb_units_max and cluster_max_generation now accept either form, with the heuristics broadcasting a scalar internally. Also warn (instead of silently truncating) when min_up_duration/ min_down_duration resolve to a non-integer number of timesteps. Co-Authored-By: Claude Sonnet 5 * Validate optim-config automatically in SimulationSession * Fix ci * Refactor e2e tests * Formatting * Refactoring * Fix solution retrieval, window size in fast heuristic bugs * Review comments * non_prop_cost in tests * Remove tests of heuristic behaviour * New mixed strategies test * Enforce heuristic-id consistency with integer-strategy * Guard bound mutation on merged relaxed/exact variables The merged Variable rebuilt for split integer/binary variables was a detached xr.concat copy: setting .lower/.upper on it silently wrote to an orphaned copy instead of the solver, and its name/label_range were inherited from only one of the two groups. Now fails loudly and carries correct metadata. Co-Authored-By: Claude Sonnet 5 * Document integer-strategy and thermal heuristics Documents the new per-component integer relaxation strategy (exact/relaxed/heuristic) and the built-in fast/accurate thermal heuristics in optim-config.md, building.md, AGENTS.md, and the changelog. Also fixes a mypy error-code annotation in optimization.py. Co-Authored-By: Claude Sonnet 5 * Update changelog * Add lower_bound()/upper_bound() expression operators Surfaces a variable's current bound in extra-outputs and port-field definitions, mirroring dual()/reduced_cost(). This lets heuristic-mutated bounds (e.g. the fast thermal heuristic tightening generation_power's lower bound) be read back post-solve instead of only the solved value. Reading bypasses the detached merged relaxed/exact variable copy the same way get_variable_solution() already does, so mutations reach the output. Use the new operator to fix num_units_on/non_prop_cost in the thermal heuristic model libraries: they previously approximated unit commitment from generation_power / max_power_per_unit, which diverges from what the fast heuristic actually enforces; deriving it from lower_bound(generation_power) / min_power_per_unit matches it exactly. Co-Authored-By: Claude Sonnet 5 * Fix tests --------- Co-authored-by: Juliette-Gerbaux Co-authored-by: Claude Sonnet 5 Co-authored-by: Thomas Bittar --- docs/CHANGELOG.md | 7 ++ docs/agents/testing.md | 2 +- src/gems_craft/expression/copy.py | 8 ++ src/gems_craft/expression/degree.py | 8 ++ src/gems_craft/expression/equality.py | 12 +++ src/gems_craft/expression/expression.py | 10 ++ src/gems_craft/expression/indexing.py | 8 ++ .../expression/parsing/parse_expression.py | 22 +++++ src/gems_craft/expression/print.py | 8 ++ .../expression/uses_sum_connections_on.py | 8 ++ src/gems_craft/expression/visitor.py | 12 +++ src/gems_craft/model/port.py | 8 ++ src/gems_craft/model/resolve_library.py | 8 ++ src/gems_runner/expression/evaluate.py | 8 ++ src/gems_runner/simulation/extra_output.py | 34 +++++++ src/gems_runner/simulation/optimization.py | 65 ++++++++++--- .../simulation/simulation_table.py | 30 ++++++ .../simulation/vectorized_builder.py | 26 ++++++ .../libs/thermal_variants_for_heuristic.yml | 4 +- ...euristic_four_clusters_mixed_strategies.py | 4 +- ...thermal_heuristic_two_clusters_low_load.py | 5 +- .../parsing/test_expression_parsing.py | 16 ++++ .../expressions/visitor/test_degree.py | 7 ++ .../expressions/visitor/test_equality.py | 17 +++- .../expressions/visitor/test_indexing.py | 17 +++- .../expressions/visitor/test_printer.py | 12 ++- .../lib_parsing/test_lib_parsing.py | 26 ++++++ .../gems_runner/expression/test_evaluation.py | 15 ++- .../simulation/simulation_table_fakes.py | 10 ++ .../simulation/test_integer_strategy.py | 59 +++++++++++- .../test_simulation_table_extra_outputs.py | 92 +++++++++++++++++++ 31 files changed, 542 insertions(+), 26 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 33e98a9c..63af7d27 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -17,6 +17,13 @@ All notable changes to GemsPy are documented here. compute tighter variable bounds from the first solve. Each model declares what a heuristic reads/writes via `models[].heuristics` in `optim-config.yml`. +- **`lower_bound(variable_name)`** and **`upper_bound(variable_name)`** operators in the + expression language, usable in `extra-outputs` and port-field-definitions. Both take a bare + variable identifier and return its *current* lower/upper bound post-solve — in particular + reflecting mutations made by thermal heuristics (see "Integer strategy and thermal + heuristics" above), which previously had no way to be surfaced in results. Validated at + model-build time; using them inside constraints, binding-constraints, objective + contributions, or variable bounds raises a `ValueError`. ### Fixed - **Standard library parsing now accepts hybrid port-type fields** - diff --git a/docs/agents/testing.md b/docs/agents/testing.md index 8db9e246..a7db98b9 100644 --- a/docs/agents/testing.md +++ b/docs/agents/testing.md @@ -23,7 +23,7 @@ nature (they build a study with `gems_craft` and solve it with `gems_runner` in | Unit — gems_craft / system | `tests/unittests/gems_craft/system/` | Model, network, and port object behaviour | | Unit — gems_craft / system parsing | `tests/unittests/gems_craft/system_parsing/` | System YAML parsing | | Unit — gems_craft_hybrid | `tests/unittests/gems_craft_hybrid/` | Hybrid GEMS/Antares Simulator schema parsing | -| Unit — gems_runner / expression | `tests/unittests/gems_runner/expression/` | Solver-output expression evaluation (`dual()`, `reduced_cost()`, `variable()`) | +| Unit — gems_runner / expression | `tests/unittests/gems_runner/expression/` | Solver-output expression evaluation (`dual()`, `reduced_cost()`, `lower_bound()`, `upper_bound()`, `variable()`) | | Unit — gems_runner / simulation | `tests/unittests/gems_runner/simulation/` | Full problem build + solve on small networks | | End-to-end — functional | `tests/e2e/functional/` | Cross-cutting tests: library/system combinations, stochastic, investment, scenario builder | | End-to-end — models | `tests/e2e/models/` | Model-level tests (andromede-v1 models, operator tests, proof-of-concept models) | diff --git a/src/gems_craft/expression/copy.py b/src/gems_craft/expression/copy.py index 90e93ff3..4e352eb5 100644 --- a/src/gems_craft/expression/copy.py +++ b/src/gems_craft/expression/copy.py @@ -23,6 +23,7 @@ ExpressionNode, FloorNode, LiteralNode, + LowerBoundNode, MaxNode, MinNode, ParameterNode, @@ -34,6 +35,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, VariableNode, ) from .visitor import ExpressionVisitorOperations, visit @@ -111,6 +113,12 @@ def dual(self, node: DualNode) -> ExpressionNode: def reduced_cost(self, node: ReducedCostNode) -> ExpressionNode: return ReducedCostNode(node.variable_id) + def lower_bound(self, node: LowerBoundNode) -> ExpressionNode: + return LowerBoundNode(node.variable_id) + + def upper_bound(self, node: UpperBoundNode) -> ExpressionNode: + return UpperBoundNode(node.variable_id) + def copy_expression(expression: ExpressionNode) -> ExpressionNode: return visit(expression, CopyVisitor()) diff --git a/src/gems_craft/expression/degree.py b/src/gems_craft/expression/degree.py index 97c007fc..4fe13ef0 100644 --- a/src/gems_craft/expression/degree.py +++ b/src/gems_craft/expression/degree.py @@ -20,6 +20,7 @@ CeilNode, DualNode, FloorNode, + LowerBoundNode, MaxNode, MinNode, PortFieldAggregatorNode, @@ -29,6 +30,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, ) from .expression import ( @@ -132,6 +134,12 @@ def dual(self, node: DualNode) -> int | float: def reduced_cost(self, node: ReducedCostNode) -> int | float: return math.inf + def lower_bound(self, node: LowerBoundNode) -> int | float: + return math.inf + + def upper_bound(self, node: UpperBoundNode) -> int | float: + return math.inf + def compute_degree(expression: ExpressionNode) -> int | float: return visit(expression, ExpressionDegreeVisitor()) diff --git a/src/gems_craft/expression/equality.py b/src/gems_craft/expression/equality.py index 1e5b356d..ede03651 100644 --- a/src/gems_craft/expression/equality.py +++ b/src/gems_craft/expression/equality.py @@ -32,6 +32,7 @@ CeilNode, DualNode, FloorNode, + LowerBoundNode, MaxNode, MinNode, PortFieldAggregatorNode, @@ -42,6 +43,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, ) @@ -115,6 +117,10 @@ def visit(self, left: ExpressionNode, right: ExpressionNode) -> bool: return self.dual(left, right) if isinstance(left, ReducedCostNode) and isinstance(right, ReducedCostNode): return self.reduced_cost(left, right) + if isinstance(left, LowerBoundNode) and isinstance(right, LowerBoundNode): + return self.lower_bound(left, right) + if isinstance(left, UpperBoundNode) and isinstance(right, UpperBoundNode): + return self.upper_bound(left, right) raise NotImplementedError(f"Equality not implemented for {left.__class__}") def literal(self, left: LiteralNode, right: LiteralNode) -> bool: @@ -217,6 +223,12 @@ def dual(self, left: DualNode, right: DualNode) -> bool: def reduced_cost(self, left: ReducedCostNode, right: ReducedCostNode) -> bool: return left.variable_id == right.variable_id + def lower_bound(self, left: LowerBoundNode, right: LowerBoundNode) -> bool: + return left.variable_id == right.variable_id + + def upper_bound(self, left: UpperBoundNode, right: UpperBoundNode) -> bool: + return left.variable_id == right.variable_id + def expressions_equal( left: ExpressionNode, right: ExpressionNode, abs_tol: float = 0, rel_tol: float = 0 diff --git a/src/gems_craft/expression/expression.py b/src/gems_craft/expression/expression.py index 00fdc9eb..45d65da2 100644 --- a/src/gems_craft/expression/expression.py +++ b/src/gems_craft/expression/expression.py @@ -357,6 +357,16 @@ class ReducedCostNode(ExpressionNode): variable_id: str +@dataclass(frozen=True, eq=False) +class LowerBoundNode(ExpressionNode): + variable_id: str + + +@dataclass(frozen=True, eq=False) +class UpperBoundNode(ExpressionNode): + variable_id: str + + def sum_expressions(expressions: Sequence[ExpressionNode]) -> ExpressionNode: if len(expressions) == 0: return LiteralNode(0) diff --git a/src/gems_craft/expression/indexing.py b/src/gems_craft/expression/indexing.py index a74d6a7c..3168b15d 100644 --- a/src/gems_craft/expression/indexing.py +++ b/src/gems_craft/expression/indexing.py @@ -27,6 +27,7 @@ ExpressionNode, FloorNode, LiteralNode, + LowerBoundNode, MaxNode, MinNode, MultiplicationNode, @@ -40,6 +41,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, VariableNode, ) from .visitor import ExpressionVisitor, T, visit @@ -156,6 +158,12 @@ def dual(self, node: DualNode) -> IndexingStructure: def reduced_cost(self, node: ReducedCostNode) -> IndexingStructure: return self.context.get_variable_structure(node.variable_id) + def lower_bound(self, node: LowerBoundNode) -> IndexingStructure: + return self.context.get_variable_structure(node.variable_id) + + def upper_bound(self, node: UpperBoundNode) -> IndexingStructure: + return self.context.get_variable_structure(node.variable_id) + def compute_indexation( expression: ExpressionNode, provider: IndexingStructureProvider diff --git a/src/gems_craft/expression/parsing/parse_expression.py b/src/gems_craft/expression/parsing/parse_expression.py index 661ab69e..cddcad6d 100644 --- a/src/gems_craft/expression/parsing/parse_expression.py +++ b/src/gems_craft/expression/parsing/parse_expression.py @@ -21,9 +21,11 @@ Comparator, ComparisonNode, DualNode, + LowerBoundNode, PortFieldAggregatorNode, PortFieldNode, ReducedCostNode, + UpperBoundNode, maximum, minimum, ) @@ -194,6 +196,22 @@ def _visit_reduced_cost(self, arg_exprs: list) -> ExpressionNode: raise ValueError(f"'{vid}' is not a variable of the model.") return ReducedCostNode(vid) + def _visit_lower_bound(self, arg_exprs: list) -> ExpressionNode: + if len(arg_exprs) != 1: + raise ValueError("lower_bound() requires exactly 1 argument.") + vid = arg_exprs[0].getText() # type: ignore + if vid not in self.identifiers.variables: + raise ValueError(f"'{vid}' is not a variable of the model.") + return LowerBoundNode(vid) + + def _visit_upper_bound(self, arg_exprs: list) -> ExpressionNode: + if len(arg_exprs) != 1: + raise ValueError("upper_bound() requires exactly 1 argument.") + vid = arg_exprs[0].getText() # type: ignore + if vid not in self.identifiers.variables: + raise ValueError(f"'{vid}' is not a variable of the model.") + return UpperBoundNode(vid) + # Visit a parse tree produced by ExprParser#function. def visitFunction(self, ctx: ExprParser.FunctionContext) -> ExpressionNode: function_name: str = ctx.IDENTIFIER().getText() # type: ignore @@ -204,6 +222,10 @@ def visitFunction(self, ctx: ExprParser.FunctionContext) -> ExpressionNode: return self._visit_dual(arg_exprs) if function_name == "reduced_cost": return self._visit_reduced_cost(arg_exprs) + if function_name == "lower_bound": + return self._visit_lower_bound(arg_exprs) + if function_name == "upper_bound": + return self._visit_upper_bound(arg_exprs) args: list[ExpressionNode] = ( [expr.accept(self) for expr in arg_exprs] # type: ignore diff --git a/src/gems_craft/expression/print.py b/src/gems_craft/expression/print.py index d90ef3c6..79ece5ce 100644 --- a/src/gems_craft/expression/print.py +++ b/src/gems_craft/expression/print.py @@ -20,6 +20,7 @@ DualNode, ExpressionNode, FloorNode, + LowerBoundNode, MaxNode, MinNode, PortFieldAggregatorNode, @@ -29,6 +30,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, ) from .expression import ( @@ -144,6 +146,12 @@ def dual(self, node: DualNode) -> str: def reduced_cost(self, node: ReducedCostNode) -> str: return f"reduced_cost({node.variable_id})" + def lower_bound(self, node: LowerBoundNode) -> str: + return f"lower_bound({node.variable_id})" + + def upper_bound(self, node: UpperBoundNode) -> str: + return f"upper_bound({node.variable_id})" + def print_expr(expression: ExpressionNode) -> str: return visit(expression, PrinterVisitor()) diff --git a/src/gems_craft/expression/uses_sum_connections_on.py b/src/gems_craft/expression/uses_sum_connections_on.py index 7824da28..0da90abe 100644 --- a/src/gems_craft/expression/uses_sum_connections_on.py +++ b/src/gems_craft/expression/uses_sum_connections_on.py @@ -21,6 +21,7 @@ ExpressionNode, FloorNode, LiteralNode, + LowerBoundNode, MaxNode, MinNode, MultiplicationNode, @@ -34,6 +35,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, VariableNode, ) @@ -122,6 +124,12 @@ def dual(self, node: DualNode) -> bool: def reduced_cost(self, node: ReducedCostNode) -> bool: return False + def lower_bound(self, node: LowerBoundNode) -> bool: + return False + + def upper_bound(self, node: UpperBoundNode) -> bool: + return False + def uses_sum_connections_on( expr: ExpressionNode, port_name: str, field_name: str diff --git a/src/gems_craft/expression/visitor.py b/src/gems_craft/expression/visitor.py index a7a15cc8..3db8c6d7 100644 --- a/src/gems_craft/expression/visitor.py +++ b/src/gems_craft/expression/visitor.py @@ -29,6 +29,7 @@ ExpressionNode, FloorNode, LiteralNode, + LowerBoundNode, MaxNode, MinNode, MultiplicationNode, @@ -42,6 +43,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, VariableNode, ) @@ -126,6 +128,12 @@ def dual(self, node: DualNode) -> T: ... @abstractmethod def reduced_cost(self, node: ReducedCostNode) -> T: ... + @abstractmethod + def lower_bound(self, node: LowerBoundNode) -> T: ... + + @abstractmethod + def upper_bound(self, node: UpperBoundNode) -> T: ... + def visit(root: ExpressionNode, visitor: ExpressionVisitor[T]) -> T: """ @@ -177,6 +185,10 @@ def visit(root: ExpressionNode, visitor: ExpressionVisitor[T]) -> T: return visitor.dual(root) elif isinstance(root, ReducedCostNode): return visitor.reduced_cost(root) + elif isinstance(root, LowerBoundNode): + return visitor.lower_bound(root) + elif isinstance(root, UpperBoundNode): + return visitor.upper_bound(root) raise ValueError(f"Unknown expression node type {root.__class__}") diff --git a/src/gems_craft/model/port.py b/src/gems_craft/model/port.py index bf6435d4..e46bf599 100644 --- a/src/gems_craft/model/port.py +++ b/src/gems_craft/model/port.py @@ -32,6 +32,7 @@ CeilNode, DualNode, FloorNode, + LowerBoundNode, MaxNode, MinNode, PortFieldAggregatorNode, @@ -42,6 +43,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, ) from gems_craft.expression.visitor import visit @@ -178,6 +180,12 @@ def dual(self, node: DualNode) -> None: def reduced_cost(self, node: ReducedCostNode) -> None: pass # reduced_cost() is permitted in port-field definitions + def lower_bound(self, node: LowerBoundNode) -> None: + pass # lower_bound() is permitted in port-field definitions + + def upper_bound(self, node: UpperBoundNode) -> None: + pass # upper_bound() is permitted in port-field definitions + def _validate_port_field_expression(definition: PortFieldDefinition) -> None: visit(definition.definition, _PortFieldExpressionChecker()) diff --git a/src/gems_craft/model/resolve_library.py b/src/gems_craft/model/resolve_library.py index 5326fe89..be06f6a6 100644 --- a/src/gems_craft/model/resolve_library.py +++ b/src/gems_craft/model/resolve_library.py @@ -23,6 +23,7 @@ DualNode, FloorNode, LiteralNode, + LowerBoundNode, MaxNode, MinNode, MultiplicationNode, @@ -36,6 +37,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, VariableNode, ) from gems_craft.expression.indexing_structure import IndexingStructure @@ -293,6 +295,12 @@ def dual(self, node: DualNode) -> None: def reduced_cost(self, node: ReducedCostNode) -> None: pass + def lower_bound(self, node: LowerBoundNode) -> None: + pass + + def upper_bound(self, node: UpperBoundNode) -> None: + pass + def _forbid_bare_port_field(expr: ExpressionNode, context: str) -> None: visit(expr, _ForbidBarePortFieldVisitor(context)) diff --git a/src/gems_runner/expression/evaluate.py b/src/gems_runner/expression/evaluate.py index c6cf5b97..0219db19 100644 --- a/src/gems_runner/expression/evaluate.py +++ b/src/gems_runner/expression/evaluate.py @@ -24,6 +24,7 @@ ExpressionNode, FloorNode, LiteralNode, + LowerBoundNode, MaxNode, MinNode, ParameterNode, @@ -35,6 +36,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, VariableNode, ) from gems_craft.expression.indexing import IndexingStructureProvider @@ -139,6 +141,12 @@ def dual(self, node: DualNode) -> float: def reduced_cost(self, node: ReducedCostNode) -> float: raise NotImplementedError("reduced_cost() is not statically evaluable.") + def lower_bound(self, node: LowerBoundNode) -> float: + raise NotImplementedError("lower_bound() is not statically evaluable.") + + def upper_bound(self, node: UpperBoundNode) -> float: + raise NotImplementedError("upper_bound() is not statically evaluable.") + def evaluate(expression: ExpressionNode, value_provider: ValueProvider) -> float: return visit(expression, EvaluationVisitor(value_provider)) diff --git a/src/gems_runner/simulation/extra_output.py b/src/gems_runner/simulation/extra_output.py index 7bc5e988..d0deb5ec 100644 --- a/src/gems_runner/simulation/extra_output.py +++ b/src/gems_runner/simulation/extra_output.py @@ -35,7 +35,9 @@ Comparator, ComparisonNode, DualNode, + LowerBoundNode, ReducedCostNode, + UpperBoundNode, VariableNode, ) from gems_craft.expression.visitor import visit @@ -110,6 +112,14 @@ class VectorizedExtraOutputBuilder(VectorizedBuilderBase[xr.DataArray]): var_reduced_cost_arrays: Mapping from (model_id, var_name) to a DataArray of reduced cost values, with dims in {component, time, scenario} (or a subset). + var_lower_bound_arrays: + Mapping from (model_id, var_name) to a DataArray of the variable's + current lower bound, with dims in {component, time, scenario} (or a + subset). + var_upper_bound_arrays: + Mapping from (model_id, var_name) to a DataArray of the variable's + current upper bound, with dims in {component, time, scenario} (or a + subset). port_arrays: Pre-computed xr.DataArray for each PortFieldId of this model. Keyed by PortFieldId(port_name, field_name). @@ -124,6 +134,12 @@ class VectorizedExtraOutputBuilder(VectorizedBuilderBase[xr.DataArray]): var_reduced_cost_arrays: Dict[Tuple[str, str], xr.DataArray] = field( default_factory=dict ) + var_lower_bound_arrays: Dict[Tuple[str, str], xr.DataArray] = field( + default_factory=dict + ) + var_upper_bound_arrays: Dict[Tuple[str, str], xr.DataArray] = field( + default_factory=dict + ) def variable(self, node: VariableNode) -> xr.DataArray: key = (self.model_id, node.name) @@ -152,6 +168,24 @@ def reduced_cost(self, node: ReducedCostNode) -> xr.DataArray: ) return self.var_reduced_cost_arrays[key] + def lower_bound(self, node: LowerBoundNode) -> xr.DataArray: + key = (self.model_id, node.variable_id) + if key not in self.var_lower_bound_arrays: + raise KeyError( + f"Lower bound of variable '{node.variable_id}' not found for model " + f"{self.model_id!r}." + ) + return self.var_lower_bound_arrays[key] + + def upper_bound(self, node: UpperBoundNode) -> xr.DataArray: + key = (self.model_id, node.variable_id) + if key not in self.var_upper_bound_arrays: + raise KeyError( + f"Upper bound of variable '{node.variable_id}' not found for model " + f"{self.model_id!r}." + ) + return self.var_upper_bound_arrays[key] + def comparison(self, node: ComparisonNode) -> xr.DataArray: """Evaluate a comparison post-solve as a float indicator DataArray. diff --git a/src/gems_runner/simulation/optimization.py b/src/gems_runner/simulation/optimization.py index 6eb4ff61..cc1905ba 100644 --- a/src/gems_runner/simulation/optimization.py +++ b/src/gems_runner/simulation/optimization.py @@ -433,18 +433,22 @@ def get_variable_labels( lv = self._linopy_vars.get((model_id, var_name)) return lv.labels if lv is not None else None - def get_variable_solution( - self, model_id: str, var_name: str + def _reassemble_variable_attr( + self, model_id: str, var_name: str, attr: str ) -> Optional[xr.DataArray]: - """Return solved values for *var_name* across all its components. - - Unlike ``linopy_model.solution[]``, this is correct even when the - variable was split across relaxed/exact strategy groups: the merged - ``_linopy_vars`` copy keeps the ``.name`` of only one of the two really - -registered group Variables, so indexing the solver's solution Dataset - by that name silently drops the other group's components. This instead - reads ``.solution`` directly off the real per-component Variables - (``_linopy_vars_by_component``) and reassembles them. + """Reassemble a per-instance ``linopy.Variable`` attribute (``solution``, + ``lower``, ``upper``) for *var_name* across all its components. + + Unlike reading the attribute off ``self._linopy_vars[(model_id, var_name)]``, + this is correct even when the variable was split across relaxed/exact + strategy groups: that merged, detached ``_MergedGroupVariable`` copy + keeps the ``.name`` of only one of the two really-registered group + Variables (so indexing the solver's solution Dataset by that name + silently drops the other group's components), and it is rebuilt once at + problem-build time so it never reflects later bound mutations (e.g. from + heuristics). This instead reads the attribute directly off the real + per-component Variables (``_linopy_vars_by_component``) and reassembles + them. """ by_name: Dict[str, linopy.Variable] = {} for (m, vn, _c), variable in self._linopy_vars_by_component.items(): @@ -456,11 +460,46 @@ def get_variable_solution( if not group_vars: return None if len(group_vars) == 1: - return group_vars[0].solution + return cast(xr.DataArray, getattr(group_vars[0], attr)) return cast( - xr.DataArray, xr.concat([v.solution for v in group_vars], dim="component") + xr.DataArray, + xr.concat([getattr(v, attr) for v in group_vars], dim="component"), ) + def get_variable_solution( + self, model_id: str, var_name: str + ) -> Optional[xr.DataArray]: + """Return solved values for *var_name* across all its components. + + See :meth:`_reassemble_variable_attr` for why this must bypass the + merged ``_linopy_vars`` copy. + """ + return self._reassemble_variable_attr(model_id, var_name, "solution") + + def get_variable_lower_bound( + self, model_id: str, var_name: str + ) -> Optional[xr.DataArray]: + """Return the current lower bound for *var_name* across all its components. + + Reflects any bound mutation applied after problem construction (e.g. by + thermal heuristics via :meth:`get_component_variable`). See + :meth:`_reassemble_variable_attr` for why this must bypass the merged + ``_linopy_vars`` copy. + """ + return self._reassemble_variable_attr(model_id, var_name, "lower") + + def get_variable_upper_bound( + self, model_id: str, var_name: str + ) -> Optional[xr.DataArray]: + """Return the current upper bound for *var_name* across all its components. + + Reflects any bound mutation applied after problem construction (e.g. by + thermal heuristics via :meth:`get_component_variable`). See + :meth:`_reassemble_variable_attr` for why this must bypass the merged + ``_linopy_vars`` copy. + """ + return self._reassemble_variable_attr(model_id, var_name, "upper") + # --------------------------------------------------------------------------- # Internal builder diff --git a/src/gems_runner/simulation/simulation_table.py b/src/gems_runner/simulation/simulation_table.py index ca993644..e3dd1559 100644 --- a/src/gems_runner/simulation/simulation_table.py +++ b/src/gems_runner/simulation/simulation_table.py @@ -279,6 +279,8 @@ def _collect_extra_outputs( constraint_dual_arrays = self._collect_constraint_duals(problem) var_reduced_cost_arrays = self._collect_reduced_costs(problem) + var_lower_bound_arrays = self._collect_lower_bounds(problem) + var_upper_bound_arrays = self._collect_upper_bounds(problem) for mk, components in problem.study.model_components.items(): model = problem.study.models[mk] @@ -295,6 +297,8 @@ def _collect_extra_outputs( var_solution_arrays=var_solution_arrays, constraint_dual_arrays=constraint_dual_arrays, var_reduced_cost_arrays=var_reduced_cost_arrays, + var_lower_bound_arrays=var_lower_bound_arrays, + var_upper_bound_arrays=var_upper_bound_arrays, port_arrays={}, block_length=problem.block_length, ), @@ -307,6 +311,8 @@ def _collect_extra_outputs( var_solution_arrays=var_solution_arrays, constraint_dual_arrays=constraint_dual_arrays, var_reduced_cost_arrays=var_reduced_cost_arrays, + var_lower_bound_arrays=var_lower_bound_arrays, + var_upper_bound_arrays=var_upper_bound_arrays, port_arrays=port_arrays, block_length=problem.block_length, ) @@ -411,6 +417,30 @@ def _collect_reduced_costs( except Exception: return {} + @staticmethod + def _collect_lower_bounds( + problem: OptimizationProblem, + ) -> Dict[Tuple[str, str], xr.DataArray]: + """Return current variable lower bounds keyed by (model_key, var_name).""" + result: Dict[Tuple[str, str], xr.DataArray] = {} + for mk, vname in problem._linopy_vars: + lb = problem.get_variable_lower_bound(mk, vname) + if lb is not None: + result[(mk, vname)] = lb + return result + + @staticmethod + def _collect_upper_bounds( + problem: OptimizationProblem, + ) -> Dict[Tuple[str, str], xr.DataArray]: + """Return current variable upper bounds keyed by (model_key, var_name).""" + result: Dict[Tuple[str, str], xr.DataArray] = {} + for mk, vname in problem._linopy_vars: + ub = problem.get_variable_upper_bound(mk, vname) + if ub is not None: + result[(mk, vname)] = ub + return result + # ------------------------------------------------------------------------- # Objective value # ------------------------------------------------------------------------- diff --git a/src/gems_runner/simulation/vectorized_builder.py b/src/gems_runner/simulation/vectorized_builder.py index 137bcc47..f6a831f6 100644 --- a/src/gems_runner/simulation/vectorized_builder.py +++ b/src/gems_runner/simulation/vectorized_builder.py @@ -51,6 +51,7 @@ ExpressionNode, FloorNode, LiteralNode, + LowerBoundNode, MaxNode, MinNode, MultiplicationNode, @@ -64,6 +65,7 @@ TimeEvalNode, TimeShiftNode, TimeSumNode, + UpperBoundNode, VariableNode, ) from gems_craft.expression.visitor import ( @@ -406,6 +408,18 @@ def reduced_cost(self, node: ReducedCostNode) -> VectorizedExpr: f"not in {type(self).__name__}." ) + def lower_bound(self, node: LowerBoundNode) -> VectorizedExpr: + raise NotImplementedError( + f"lower_bound() is only available in the extra-output builder, " + f"not in {type(self).__name__}." + ) + + def upper_bound(self, node: UpperBoundNode) -> VectorizedExpr: + raise NotImplementedError( + f"upper_bound() is only available in the extra-output builder, " + f"not in {type(self).__name__}." + ) + # ------------------------------------------------------------------ # # Private helpers # # ------------------------------------------------------------------ # @@ -573,6 +587,12 @@ def dual(self, node: DualNode) -> xr.DataArray: def reduced_cost(self, node: ReducedCostNode) -> xr.DataArray: raise NotImplementedError + def lower_bound(self, node: LowerBoundNode) -> xr.DataArray: + raise NotImplementedError + + def upper_bound(self, node: UpperBoundNode) -> xr.DataArray: + raise NotImplementedError + def _and_mask( a: Optional[xr.DataArray], b: Optional[xr.DataArray] @@ -741,3 +761,9 @@ def dual(self, node: DualNode) -> Optional[xr.DataArray]: def reduced_cost(self, node: ReducedCostNode) -> Optional[xr.DataArray]: return None + + def lower_bound(self, node: LowerBoundNode) -> Optional[xr.DataArray]: + return None + + def upper_bound(self, node: UpperBoundNode) -> Optional[xr.DataArray]: + return None diff --git a/tests/e2e/functional/libs/thermal_variants_for_heuristic.yml b/tests/e2e/functional/libs/thermal_variants_for_heuristic.yml index 95e5d8de..bd7011c2 100644 --- a/tests/e2e/functional/libs/thermal_variants_for_heuristic.yml +++ b/tests/e2e/functional/libs/thermal_variants_for_heuristic.yml @@ -154,6 +154,6 @@ library: expression: expec(sum(market_bid_cost * generation_power)) extra-outputs: - id: num_units_on - expression: ceil(generation_power / max_power_per_unit) + expression: max(floor(lower_bound(generation_power) / min_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))) - id: non_prop_cost - expression: startup_cost * max(0,ceil(generation_power / max_power_per_unit)-(ceil(generation_power / max_power_per_unit))[t-1])+ fixed_cost * ceil(generation_power / max_power_per_unit) + expression: startup_cost * max(0,max(floor(lower_bound(generation_power) / min_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100)))-(max(floor(lower_bound(generation_power) / min_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))))[t-1])+ fixed_cost * max(floor(lower_bound(generation_power) / min_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))) diff --git a/tests/e2e/functional/test_thermal_heuristic_four_clusters_mixed_strategies.py b/tests/e2e/functional/test_thermal_heuristic_four_clusters_mixed_strategies.py index 00ceebcb..b8c5a97c 100644 --- a/tests/e2e/functional/test_thermal_heuristic_four_clusters_mixed_strategies.py +++ b/tests/e2e/functional/test_thermal_heuristic_four_clusters_mixed_strategies.py @@ -55,7 +55,7 @@ _G3_GEN = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 100.0, 328.0, 503.0, 563.0, 600.0, 563.0, 503.0, 328.0, 100.0, 0.0, 0.0, 0.0, 0.0] _G4_GEN = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 50.0, 50.0, 50.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] -_G2_NODU = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 1.0, 1.0] +_G2_NODU = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0] _G3_NODU = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 1.0, 0.0, 0.0, 0.0, 0.0] _G4_NODU = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] # fmt: on @@ -90,5 +90,5 @@ def test_mixed_integer_strategies() -> None: check_output(st, "N", "spilled_energy", [0.0] * 24) assert total_output_sum(st, THERMAL_COMPONENTS, "non_prop_cost") == pytest.approx( - 13662 + 13665 ) diff --git a/tests/e2e/functional/test_thermal_heuristic_two_clusters_low_load.py b/tests/e2e/functional/test_thermal_heuristic_two_clusters_low_load.py index 6ae6c433..dd641726 100644 --- a/tests/e2e/functional/test_thermal_heuristic_two_clusters_low_load.py +++ b/tests/e2e/functional/test_thermal_heuristic_two_clusters_low_load.py @@ -53,7 +53,7 @@ _G1_GEN_FAST = [24600.0, 24600.0, 24600.0, 24600.0, 24600.0, 24600.0, 29152.0, 28840.0, 30246.0, 29912.0, 27437.0, 31543.0, 38677.0, 36685.0, 31115.0, 22643.0, 17874.0, 18589.0, 19041.0, 16725.0, 15600.0, 15600.0, 15600.0, 6724.0, 4800.0, 4800.0, 4800.0, 4800.0, 4800.0, 4800.0, 4800.0, 9000.0, 9000.0, 9000.0, 9000.0, 9000.0, 13147.0, 12872.0, 9425.0, 10200.0, 10200.0, 10200.0, 11457.0, 11380.0, 12093.0, 12033.0, 14753.0, 12000.0, 12000.0, 12827.0, 12000.0, 12000.0, 12000.0, 12172.0, 17738.0, 19634.0, 19837.0, 17800.0, 13800.0, 16393.0, 19334.0, 17565.0, 14124.0, 14137.0, 13316.0, 16819.0, 19358.0, 15619.0, 13200.0, 13200.0, 13200.0, 6600.0, 6600.0, 6600.0, 6600.0, 6600.0, 6600.0, 6600.0, 9673.0, 15000.0, 15000.0, 15000.0, 15000.0, 16806.0, 21925.0, 20016.0, 16679.0, 14606.0, 13176.0, 15619.0, 17136.0, 14818.0, 12000.0, 12000.0, 12000.0, 5400.0, 5400.0, 5400.0, 5400.0, 5400.0, 5400.0, 5400.0, 7893.0, 18000.0, 18000.0, 18000.0, 18000.0, 21118.0, 26935.0, 26345.0, 24368.0, 22581.0, 24429.0, 31131.0, 31414.0, 28160.0, 24430.0, 22302.0, 23479.0, 20638.0, 14400.0, 14400.0, 14400.0, 14400.0, 14400.0, 14400.0, 21222.0, 28128.0, 29724.0, 29975.0, 24316.0, 27662.0, 32183.0, 30524.0, 30265.0, 31729.0, 35121.0, 40762.0, 44185.0, 45000.0, 41891.0, 41048.0, 41509.0, 39384.0, 34533.0, 34661.0, 33049.0, 31777.0, 31616.0, 31693.0, 38073.0, 39023.0, 38405.0, 36189.0, 30393.0, 35535.0, 42843.0, 43891.0, 42826.0, 43974.0, 45000.0, 45000.0, 45000.0, 45000.0, 42494.0, 38634.0, 39132.0, 36502.0] _G2_GEN_FAST = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 67.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1937.0, 5358.0, 5539.0, 3045.0, 0.0, 0.0, 0.0, 0.0] -_G1_NODU_FAST = [41.0, 41.0, 41.0, 41.0, 41.0, 41.0, 41.0, 43.0, 43.0, 43.0, 43.0, 43.0, 43.0, 43.0, 43.0, 26.0, 26.0, 26.0, 26.0, 26.0, 26.0, 26.0, 26.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 15.0, 15.0, 15.0, 15.0, 15.0, 15.0, 15.0, 15.0, 17.0, 17.0, 17.0, 17.0, 17.0, 17.0, 17.0, 17.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 23.0, 23.0, 23.0, 23.0, 23.0, 23.0, 23.0, 23.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0, 11.0, 11.0, 11.0, 11.0, 11.0, 11.0, 11.0, 11.0, 25.0, 25.0, 25.0, 25.0, 25.0, 25.0, 25.0, 25.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 9.0, 9.0, 9.0, 9.0, 9.0, 9.0, 9.0, 9.0, 30.0, 30.0, 30.0, 30.0, 30.0, 30.0, 30.0, 30.0, 35.0, 35.0, 35.0, 35.0, 35.0, 35.0, 35.0, 35.0, 24.0, 24.0, 24.0, 24.0, 24.0, 24.0, 24.0, 24.0, 36.0, 36.0, 36.0, 36.0, 36.0, 36.0, 36.0, 36.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 41.0] +_G1_NODU_FAST = [41.0, 41.0, 41.0, 41.0, 41.0, 41.0, 41.0, 43.0, 43.0, 43.0, 43.0, 43.0, 43.0, 43.0, 43.0, 26.0, 26.0, 26.0, 26.0, 26.0, 26.0, 26.0, 26.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 15.0, 15.0, 15.0, 15.0, 15.0, 15.0, 15.0, 15.0, 17.0, 17.0, 17.0, 17.0, 17.0, 17.0, 17.0, 17.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 23.0, 23.0, 23.0, 23.0, 23.0, 23.0, 23.0, 23.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0, 11.0, 11.0, 11.0, 11.0, 11.0, 11.0, 11.0, 11.0, 25.0, 25.0, 25.0, 25.0, 25.0, 25.0, 25.0, 25.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 9.0, 9.0, 9.0, 9.0, 9.0, 9.0, 9.0, 9.0, 30.0, 30.0, 30.0, 30.0, 30.0, 30.0, 30.0, 30.0, 35.0, 35.0, 35.0, 35.0, 35.0, 35.0, 35.0, 35.0, 24.0, 24.0, 24.0, 24.0, 24.0, 24.0, 24.0, 24.0, 36.0, 36.0, 36.0, 36.0, 36.0, 36.0, 36.0, 36.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 44.0, 44.0, 44.0, 44.0, 44.0, 44.0, 44.0, 44.0, 49.0, 49.0, 49.0, 49.0, 49.0, 49.0, 49.0, 49.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 50.0, 41.0] _G2_NODU_FAST = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 6.0, 6.0, 4.0, 0.0, 0.0, 0.0, 0.0] _SPIL_FAST = [4811.0, 4163.0, 4503.0, 2473.0, 697.0, 800.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3102.0, 4107.0, 4344.0, 0.0, 4491.0, 4673.0, 4798.0, 4706.0, 4339.0, 4283.0, 4079.0, 8019.0, 7483.0, 6327.0, 5720.0, 958.0, 0.0, 0.0, 0.0, 3930.0, 4173.0, 400.0, 0.0, 0.0, 0.0, 0.0, 0.0, 939.0, 333.0, 0.0, 1071.0, 1791.0, 2358.0, 0.0, 0.0, 0.0, 0.0, 0.0, 397.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3562.0, 7636.0, 7958.0, 4913.0, 5642.0, 6156.0, 6502.0, 6368.0, 6169.0, 4857.0, 0.0, 1405.0, 1373.0, 1983.0, 3635.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 826.0, 3465.0, 4101.0, 1176.0, 3191.0, 4589.0, 5003.0, 4510.0, 4611.0, 4778.0, 0.0, 4926.0, 4174.0, 3533.0, 5550.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3122.0, 4855.0, 7280.0, 7215.0, 5066.0, 1475.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] _UNSP_FAST = [0.0] * 168 @@ -106,7 +106,6 @@ def test_accurate_heuristic() -> None: 2641690 ) -@pytest.mark.xfail(reason="Computation of num_units_on with fast heuristic not implemented.") def test_fast_heuristic() -> None: """ Solve the same problem with the fast heuristic. @@ -127,5 +126,5 @@ def test_fast_heuristic() -> None: check_output(st, "N", "spilled_energy", _SPIL_FAST) assert sum(_SPIL_FAST) == pytest.approx(255873) assert total_output_sum(st, THERMAL_COMPONENTS, "non_prop_cost") == pytest.approx( - 4640150 + 2273100 ) diff --git a/tests/unittests/gems_craft/expressions/parsing/test_expression_parsing.py b/tests/unittests/gems_craft/expressions/parsing/test_expression_parsing.py index ee430edf..a8c253ec 100644 --- a/tests/unittests/gems_craft/expressions/parsing/test_expression_parsing.py +++ b/tests/unittests/gems_craft/expressions/parsing/test_expression_parsing.py @@ -17,7 +17,9 @@ from gems_craft.expression.equality import expressions_equal from gems_craft.expression.expression import ( DualNode, + LowerBoundNode, ReducedCostNode, + UpperBoundNode, maximum, minimum, port_field, @@ -251,6 +253,8 @@ def test_parsing_visitor( [ (set(), set(), {"balance"}, "dual(balance)", DualNode("balance")), ({"p"}, set(), set(), "reduced_cost(p)", ReducedCostNode("p")), + ({"p"}, set(), set(), "lower_bound(p)", LowerBoundNode("p")), + ({"p"}, set(), set(), "upper_bound(p)", UpperBoundNode("p")), ], ) def test_parsing_dual_and_reduced_cost( @@ -277,6 +281,18 @@ def test_parse_reduced_cost_unknown_variable_raises() -> None: parse_expression("reduced_cost(p)", identifiers) +def test_parse_lower_bound_unknown_variable_raises() -> None: + identifiers = ModelIdentifiers({"x"}, set(), set()) + with pytest.raises(ParsingException, match="not a variable"): + parse_expression("lower_bound(p)", identifiers) + + +def test_parse_upper_bound_unknown_variable_raises() -> None: + identifiers = ModelIdentifiers({"x"}, set(), set()) + with pytest.raises(ParsingException, match="not a variable"): + parse_expression("upper_bound(p)", identifiers) + + @pytest.mark.parametrize( "expression_str", [ diff --git a/tests/unittests/gems_craft/expressions/visitor/test_degree.py b/tests/unittests/gems_craft/expressions/visitor/test_degree.py index 0bceccea..0ceb958c 100644 --- a/tests/unittests/gems_craft/expressions/visitor/test_degree.py +++ b/tests/unittests/gems_craft/expressions/visitor/test_degree.py @@ -28,8 +28,10 @@ CeilNode, DualNode, FloorNode, + LowerBoundNode, ReducedCostNode, RoundNode, + UpperBoundNode, ) @@ -87,6 +89,11 @@ def test_dual_reduced_cost_degree() -> None: assert visit(ReducedCostNode("p"), ExpressionDegreeVisitor()) == math.inf +def test_lower_upper_bound_degree() -> None: + assert visit(LowerBoundNode("x"), ExpressionDegreeVisitor()) == math.inf + assert visit(UpperBoundNode("x"), ExpressionDegreeVisitor()) == math.inf + + @pytest.mark.xfail(reason="Degree simplification not implemented") def test_degree_computation_should_take_into_account_simplifications() -> None: x = var("x") diff --git a/tests/unittests/gems_craft/expressions/visitor/test_equality.py b/tests/unittests/gems_craft/expressions/visitor/test_equality.py index 0c831a73..9fdae1d4 100644 --- a/tests/unittests/gems_craft/expressions/visitor/test_equality.py +++ b/tests/unittests/gems_craft/expressions/visitor/test_equality.py @@ -14,7 +14,14 @@ from gems_craft.expression import ExpressionNode, copy_expression, literal, param, var from gems_craft.expression.equality import expressions_equal -from gems_craft.expression.expression import DualNode, ReducedCostNode, maximum, minimum +from gems_craft.expression.expression import ( + DualNode, + LowerBoundNode, + ReducedCostNode, + UpperBoundNode, + maximum, + minimum, +) @pytest.mark.parametrize( @@ -94,3 +101,11 @@ def test_dual_reduced_cost_equality() -> None: ) assert not expressions_equal(ReducedCostNode("p"), ReducedCostNode("q")) assert not expressions_equal(DualNode("p"), ReducedCostNode("p")) + + +def test_lower_upper_bound_equality() -> None: + assert expressions_equal(LowerBoundNode("x"), copy_expression(LowerBoundNode("x"))) + assert not expressions_equal(LowerBoundNode("x"), LowerBoundNode("y")) + assert expressions_equal(UpperBoundNode("x"), copy_expression(UpperBoundNode("x"))) + assert not expressions_equal(UpperBoundNode("x"), UpperBoundNode("y")) + assert not expressions_equal(LowerBoundNode("x"), UpperBoundNode("x")) diff --git a/tests/unittests/gems_craft/expressions/visitor/test_indexing.py b/tests/unittests/gems_craft/expressions/visitor/test_indexing.py index bf6aa7fa..84e94943 100644 --- a/tests/unittests/gems_craft/expressions/visitor/test_indexing.py +++ b/tests/unittests/gems_craft/expressions/visitor/test_indexing.py @@ -12,7 +12,12 @@ from gems_craft.expression import param, var -from gems_craft.expression.expression import DualNode, ReducedCostNode +from gems_craft.expression.expression import ( + DualNode, + LowerBoundNode, + ReducedCostNode, + UpperBoundNode, +) from gems_craft.expression.indexing import IndexingStructureProvider, compute_indexation from gems_craft.expression.indexing_structure import IndexingStructure @@ -122,3 +127,13 @@ def test_dual_reduced_cost_indexing() -> None: assert compute_indexation(ReducedCostNode("p"), provider) == IndexingStructure( True, True ) + + +def test_lower_upper_bound_indexing() -> None: + provider = StructureProvider() + assert compute_indexation(LowerBoundNode("x"), provider) == IndexingStructure( + True, True + ) + assert compute_indexation(UpperBoundNode("x"), provider) == IndexingStructure( + True, True + ) diff --git a/tests/unittests/gems_craft/expressions/visitor/test_printer.py b/tests/unittests/gems_craft/expressions/visitor/test_printer.py index 2b6d71de..37d406e7 100644 --- a/tests/unittests/gems_craft/expressions/visitor/test_printer.py +++ b/tests/unittests/gems_craft/expressions/visitor/test_printer.py @@ -11,7 +11,12 @@ # This file is part of the Antares project. from gems_craft.expression import ExpressionNode, PrinterVisitor, param, var, visit -from gems_craft.expression.expression import DualNode, ReducedCostNode +from gems_craft.expression.expression import ( + DualNode, + LowerBoundNode, + ReducedCostNode, + UpperBoundNode, +) def test_comparison() -> None: @@ -55,3 +60,8 @@ def test_abs_round_printer() -> None: def test_dual_reduced_cost_printer() -> None: assert visit(DualNode("balance"), PrinterVisitor()) == "dual(balance)" assert visit(ReducedCostNode("p"), PrinterVisitor()) == "reduced_cost(p)" + + +def test_lower_upper_bound_printer() -> None: + assert visit(LowerBoundNode("x"), PrinterVisitor()) == "lower_bound(x)" + assert visit(UpperBoundNode("x"), PrinterVisitor()) == "upper_bound(x)" diff --git a/tests/unittests/gems_craft/lib_parsing/test_lib_parsing.py b/tests/unittests/gems_craft/lib_parsing/test_lib_parsing.py index e3930fb8..a2c9e08f 100644 --- a/tests/unittests/gems_craft/lib_parsing/test_lib_parsing.py +++ b/tests/unittests/gems_craft/lib_parsing/test_lib_parsing.py @@ -21,7 +21,9 @@ ) from gems_craft.expression.expression import ( DualNode, + LowerBoundNode, ReducedCostNode, + UpperBoundNode, maximum, minimum, port_field, @@ -307,6 +309,8 @@ def test_reduced_cost_in_objective_is_rejected() -> None: _UNRESTRICTED_EXPRS = [ pytest.param("dual(balance)", DualNode("balance"), id="dual"), pytest.param("reduced_cost(x)", ReducedCostNode("x"), id="reduced_cost"), + pytest.param("lower_bound(x)", LowerBoundNode("x"), id="lower_bound"), + pytest.param("upper_bound(x)", UpperBoundNode("x"), id="upper_bound"), pytest.param("max(x, y)", maximum(var("x"), var("y")), id="max"), pytest.param("min(x, y)", minimum(var("x"), var("y")), id="min"), pytest.param("abs(x)", var("x").abs(), id="abs"), @@ -654,3 +658,25 @@ def test_sum_connections_on_non_own_port_accepted() -> None: ) ) resolve_library([input_lib]) # must not raise + + +# --------------------------------------------------------------------------- +# Acceptance: lower_bound()/upper_bound() parse from YAML inside extra-outputs +# --------------------------------------------------------------------------- + + +def test_lower_upper_bound_in_extra_output_accepted() -> None: + """lower_bound(generation)/upper_bound(generation) are valid extra-output expressions.""" + input_lib = parse_yaml_library( + io.StringIO( + _port_model_yaml( + extra_output_expr="lower_bound(generation) + upper_bound(generation)" + ) + ) + ) + lib = resolve_library([input_lib]) + eo = lib["test"].models["test.gen_model"].extra_outputs + assert eo is not None + assert expressions_equal( + eo["eo"], LowerBoundNode("generation") + UpperBoundNode("generation") + ) diff --git a/tests/unittests/gems_runner/expression/test_evaluation.py b/tests/unittests/gems_runner/expression/test_evaluation.py index 72feee68..785ecb9b 100644 --- a/tests/unittests/gems_runner/expression/test_evaluation.py +++ b/tests/unittests/gems_runner/expression/test_evaluation.py @@ -27,7 +27,12 @@ visit, ) from gems_craft.expression.equality import expressions_equal -from gems_craft.expression.expression import DualNode, ReducedCostNode +from gems_craft.expression.expression import ( + DualNode, + LowerBoundNode, + ReducedCostNode, + UpperBoundNode, +) from gems_runner.expression import EvaluationContext, EvaluationVisitor, ValueProvider @@ -110,3 +115,11 @@ def test_dual_reduced_cost_evaluation_raises() -> None: visit(DualNode("balance"), EvaluationVisitor(ctx)) with pytest.raises(NotImplementedError, match="reduced_cost"): visit(ReducedCostNode("p"), EvaluationVisitor(ctx)) + + +def test_lower_upper_bound_evaluation_raises() -> None: + ctx = EvaluationContext() + with pytest.raises(NotImplementedError, match="lower_bound"): + visit(LowerBoundNode("x"), EvaluationVisitor(ctx)) + with pytest.raises(NotImplementedError, match="upper_bound"): + visit(UpperBoundNode("x"), EvaluationVisitor(ctx)) diff --git a/tests/unittests/gems_runner/simulation/simulation_table_fakes.py b/tests/unittests/gems_runner/simulation/simulation_table_fakes.py index 89027647..fea1a83b 100644 --- a/tests/unittests/gems_runner/simulation/simulation_table_fakes.py +++ b/tests/unittests/gems_runner/simulation/simulation_table_fakes.py @@ -78,6 +78,16 @@ def get_variable_solution( return None return self.linopy_model.solution.get(lv.name) + def get_variable_lower_bound( + self, model_id: object, var_name: str + ) -> Optional[xr.DataArray]: + return None + + def get_variable_upper_bound( + self, model_id: object, var_name: str + ) -> Optional[xr.DataArray]: + return None + def to_object_dtype(frame: pd.DataFrame) -> pd.DataFrame: """Cast every column to numpy object dtype, normalising all nulls to None.""" diff --git a/tests/unittests/gems_runner/simulation/test_integer_strategy.py b/tests/unittests/gems_runner/simulation/test_integer_strategy.py index e9432ea1..093a1459 100644 --- a/tests/unittests/gems_runner/simulation/test_integer_strategy.py +++ b/tests/unittests/gems_runner/simulation/test_integer_strategy.py @@ -19,8 +19,9 @@ """ import pandas as pd +import pytest -from gems_craft.expression.expression import literal, param +from gems_craft.expression.expression import LowerBoundNode, literal, param, var from gems_craft.expression.indexing_structure import IndexingStructure from gems_craft.model.model import model from gems_craft.model.parameter import float_parameter @@ -30,6 +31,7 @@ from gems_craft.study.parsing import HeuristicId, IntegerStrategy, IntegerStrategyId from gems_craft.study.system import Component from gems_runner.simulation import TimeBlock, build_problem +from gems_runner.simulation.simulation_table import SimulationTableBuilder MIXED_MODEL = model( id="mixed_model", @@ -179,3 +181,58 @@ def test_mixed_strategies_with_time_dependent_parameter_bound() -> None: assert ( relaxed_var is not None and relaxed_var.upper.sel(component="c2").item() == 10.0 ) + + +MIXED_MODEL_WITH_BOUND_OUTPUT = model( + id="mixed_model_with_bound_output", + variables=[ + float_variable("generation", lower_bound=literal(0), upper_bound=literal(100)), + ], + extra_outputs={ + "gen_lb": LowerBoundNode("generation"), + }, + objective_contributions={ + "null_objective": (literal(0) * var("generation")).time_sum().expec() + }, +) + + +def test_lower_bound_extra_output_bypasses_merged_group_variable() -> None: + """lower_bound() extra-outputs must read the real per-component linopy + Variable, not the merged/detached _MergedGroupVariable copy built for + models split across relaxed/exact strategy groups — otherwise a + heuristic-style bound mutation on one component would not be visible (or + would leak across components). + """ + system = System("test") + for comp_id, strategy in zip( + ["c1", "c2"], [IntegerStrategyId.EXACT, IntegerStrategyId.RELAXED] + ): + system.add_component( + Component( + model=MIXED_MODEL_WITH_BOUND_OUTPUT, + id=comp_id, + integer_strategy=IntegerStrategy(id=strategy), + ) + ) + + problem = build_problem( + Study(system, DataBase()), TimeBlock(1, [0]), scenario_ids=[0] + ) + problem.solve(solver_name="highs") + + # Simulate what a heuristic does: mutate c2's bound directly via the real + # per-component Variable, bypassing the merged/detached copy. + c2_var = problem.get_component_variable( + "mixed_model_with_bound_output", "generation", "c2" + ) + assert c2_var is not None + c2_var.lower.sel(component="c2")[:] = 42.0 + + st = SimulationTableBuilder().build(problem) + + c2_lb = st.component("c2").output("gen_lb").value(time_index=0, scenario_index=0) + c1_lb = st.component("c1").output("gen_lb").value(time_index=0, scenario_index=0) + + assert c2_lb == pytest.approx(42.0) + assert c1_lb == pytest.approx(0.0) diff --git a/tests/unittests/gems_runner/simulation/test_simulation_table_extra_outputs.py b/tests/unittests/gems_runner/simulation/test_simulation_table_extra_outputs.py index f48aebd5..6f81ce12 100644 --- a/tests/unittests/gems_runner/simulation/test_simulation_table_extra_outputs.py +++ b/tests/unittests/gems_runner/simulation/test_simulation_table_extra_outputs.py @@ -241,6 +241,98 @@ def test_extra_output_min_on_variable() -> None: assert capped == pytest.approx(3.0) +def test_extra_output_lower_bound_and_upper_bound() -> None: + """ + lower_bound(x)/upper_bound(x) return the variable's current bounds post-solve. + """ + from gems_craft.expression.expression import ( + LowerBoundNode, + UpperBoundNode, + literal, + var, + ) + from gems_craft.model.model import model + from gems_craft.model.variable import float_variable + from gems_craft.study import DataBase, Study, System, create_component + from gems_runner.simulation import TimeBlock, build_problem + + SIMPLE_MODEL = model( + id="SIMPLE_BOUNDS", + variables=[float_variable("x", lower_bound=literal(2), upper_bound=literal(7))], + extra_outputs={ + "lb": LowerBoundNode("x"), + "ub": UpperBoundNode("x"), + }, + objective_contributions={ + "null_objective": (literal(0) * var("x")).time_sum().expec() + }, + ) + + database = DataBase() + comp = create_component(model=SIMPLE_MODEL, id="comp_1") + + system = System("test_bounds_extra") + system.add_component(comp) + + problem = build_problem( + Study(system, database), TimeBlock(1, [0]), scenario_ids=list(range(1)) + ) + problem.solve(solver_name="highs") + + df = SimulationTableBuilder().build(problem) + lb = df.component("comp_1").output("lb").value(time_index=0, scenario_index=0) + ub = df.component("comp_1").output("ub").value(time_index=0, scenario_index=0) + assert lb == pytest.approx(2.0) + assert ub == pytest.approx(7.0) + + +def test_extra_output_bound_broadcasts_over_time() -> None: + """ + upper_bound(x), when x's bound is a constant parameter, is broadcast over + every timestep of a time-varying variable in the resulting SimulationTable + (not just t=0). + """ + from gems_craft.expression import param + from gems_craft.expression.expression import UpperBoundNode, literal, var + from gems_craft.expression.indexing_structure import IndexingStructure + from gems_craft.model.model import model + from gems_craft.model.parameter import float_parameter + from gems_craft.model.variable import float_variable + from gems_craft.study import ConstantData, DataBase, Study, System, create_component + from gems_runner.simulation import TimeBlock, build_problem + + SIMPLE_MODEL = model( + id="SIMPLE_BOUND_BROADCAST", + parameters=[float_parameter("cap", structure=IndexingStructure(False, False))], + variables=[ + float_variable("x", lower_bound=literal(0), upper_bound=param("cap")) + ], + extra_outputs={"x_ub": UpperBoundNode("x")}, + objective_contributions={ + "null_objective": (literal(0) * var("x")).time_sum().expec() + }, + ) + + database = DataBase() + comp = create_component(model=SIMPLE_MODEL, id="comp_1") + database.add_data("comp_1", "cap", ConstantData(7.0)) + + system = System("test_bound_broadcast") + system.add_component(comp) + + problem = build_problem( + Study(system, database), TimeBlock(1, [0, 1, 2]), scenario_ids=list(range(1)) + ) + problem.solve(solver_name="highs") + + df = SimulationTableBuilder().build(problem) + for t in range(3): + x_ub = ( + df.component("comp_1").output("x_ub").value(time_index=t, scenario_index=0) + ) + assert x_ub == pytest.approx(7.0), f"x_ub at t={t}: expected 7.0, got {x_ub}" + + def test_extra_output_comparison() -> None: """ Comparison operators (>=, <=) are allowed in extra outputs and evaluated From 82ee0c398e08b692c9e86fcc009b52c01ba491b5 Mon Sep 17 00:00:00 2001 From: Juliette-Gerbaux <130555142+Juliette-Gerbaux@users.noreply.github.com> Date: Mon, 24 Aug 2026 10:32:43 +0200 Subject: [PATCH 2/4] Fix/fast heuristic with no pmin (#284) * New test with no pmin * Fix fast heuristic to track upper bound when min_power_per_unit is 0 Previously find_min_generation_fast short-circuited to all zeros whenever min_power_per_unit was ~0, dropping the unit-commitment info needed to keep committed units available for production. It now always computes num_units_on and returns both a minimum_generation_power and a maximum_generation_power series, with the upper bound wired through the heuristic outputs schema and the one-cluster no-pmin study. --- docs/user-guide/optim-config.md | 2 +- src/gems_craft/optim_config/parsing.py | 2 +- src/gems_craft/optim_config/validation.py | 1 + .../simulation/heuristic_runner.py | 28 ++- .../simulation/thermal_heuristic.py | 28 +-- .../libs/thermal_variants_for_heuristic.yml | 4 +- .../thermal_variants_for_heuristic.yml | 3 + .../data-series/demand-ts-only-hour13.txt | 168 ++++++++++++++++ ...t_thermal_heuristic_one_cluster_no_pmin.py | 181 ++++++++++++++++++ .../functional/thermal_heuristic_helpers.py | 30 ++- .../optim_config/test_heuristic_validation.py | 5 + .../simulation/test_thermal_heuristic.py | 20 +- 12 files changed, 447 insertions(+), 25 deletions(-) create mode 100644 tests/e2e/functional/studies/thermal_heuristic_one_cluster/input/data-series/demand-ts-only-hour13.txt create mode 100644 tests/e2e/functional/test_thermal_heuristic_one_cluster_no_pmin.py diff --git a/docs/user-guide/optim-config.md b/docs/user-guide/optim-config.md index 9cc8ef82..0111c6c4 100644 --- a/docs/user-guide/optim-config.md +++ b/docs/user-guide/optim-config.md @@ -417,7 +417,7 @@ Two heuristics are built in, each expecting a fixed set of | Heuristic | `inputs` elements | `outputs` elements | |---|---|---| -| `fast` | `generation_power`, `cluster_max_generation`, `min_power_per_unit`, `max_power_per_unit`, `min_up_duration`, `min_down_duration` | `minimum_generation_power` | +| `fast` | `generation_power`, `cluster_max_generation`, `min_power_per_unit`, `max_power_per_unit`, `min_up_duration`, `min_down_duration` | `minimum_generation_power`, `maximum_generation_power` | | `accurate` | `num_units_on_opt`, `num_units_max`, `min_up_duration`, `min_down_duration` | `minimum_num_units_on` | !!! note diff --git a/src/gems_craft/optim_config/parsing.py b/src/gems_craft/optim_config/parsing.py index 69631e44..fcf6ae40 100644 --- a/src/gems_craft/optim_config/parsing.py +++ b/src/gems_craft/optim_config/parsing.py @@ -93,7 +93,7 @@ class HeuristicElementConfig(ModifiedBaseModel): "min_up_duration", "min_down_duration", }, - "outputs": {"minimum_generation_power"}, + "outputs": {"minimum_generation_power", "maximum_generation_power"}, }, } diff --git a/src/gems_craft/optim_config/validation.py b/src/gems_craft/optim_config/validation.py index 65c68d00..a4cc3624 100644 --- a/src/gems_craft/optim_config/validation.py +++ b/src/gems_craft/optim_config/validation.py @@ -56,6 +56,7 @@ "generation_power": True, "minimum_num_units_on": True, "minimum_generation_power": True, + "maximum_generation_power": True, "num_units_max": None, "cluster_max_generation": None, } diff --git a/src/gems_runner/simulation/heuristic_runner.py b/src/gems_runner/simulation/heuristic_runner.py index 7566bec7..fe901b24 100644 --- a/src/gems_runner/simulation/heuristic_runner.py +++ b/src/gems_runner/simulation/heuristic_runner.py @@ -39,11 +39,19 @@ from gems_craft.study.study import Study from gems_runner.simulation.optimization import OptimizationProblem -_HEURISTIC_FUNCTIONS: Dict[HeuristicId, Callable[..., List]] = { +_HEURISTIC_FUNCTIONS: Dict[HeuristicId, Callable[..., Any]] = { HeuristicId.FAST: find_min_generation_fast, HeuristicId.ACCURATE: find_num_units_accurate, } +# Canonical order of the heuristic_element names in each function's return value. +# Needed because HeuristicConfig.outputs is ordered by the user's YAML declaration, which +# must not be assumed to match the function's return order. +_HEURISTIC_OUTPUT_ORDER: Dict[HeuristicId, List[str]] = { + HeuristicId.FAST: ["minimum_generation_power", "maximum_generation_power"], + HeuristicId.ACCURATE: ["minimum_num_units_on"], +} + def should_apply_heuristics(study: "Study") -> bool: """Return True when the two-pass heuristic solve is needed.""" @@ -194,15 +202,23 @@ def apply_thermal_heuristics( } if "solver_name" in fn_params: kwargs["solver_name"] = solver_name - heuristic_result: List = heuristic_fn(**kwargs) # type: ignore[call-overload] + heuristic_result = heuristic_fn(**kwargs) # type: ignore[call-overload] - result_da = xr.DataArray( - heuristic_result, - dims=["time"], - coords={"time": list(range(len(heuristic_result)))}, + output_order = _HEURISTIC_OUTPUT_ORDER[heuristic_config.id] + result_values = ( + heuristic_result + if isinstance(heuristic_result, tuple) + else [heuristic_result] ) + results_by_element = dict(zip(output_order, result_values)) for output in heuristic_config.outputs: + result_list = results_by_element[output.heuristic_element] + result_da = xr.DataArray( + result_list, + dims=["time"], + coords={"time": list(range(len(result_list)))}, + ) linopy_var = _get_component_linopy_var( problem, model_id, output.id, component.id ) diff --git a/src/gems_runner/simulation/thermal_heuristic.py b/src/gems_runner/simulation/thermal_heuristic.py index 36a12e1a..3286e849 100644 --- a/src/gems_runner/simulation/thermal_heuristic.py +++ b/src/gems_runner/simulation/thermal_heuristic.py @@ -12,7 +12,7 @@ import math import warnings -from typing import List, Union +from typing import List, Tuple, Union import linopy import numpy as np @@ -78,7 +78,7 @@ def find_min_generation_fast( max_power_per_unit: float, min_up_duration: float, min_down_duration: float, -) -> List[float]: +) -> Tuple[List[float], List[float]]: """ Fast heuristic: derives the minimum generation power from the optimised production timeseries. Timesteps are grouped into windows of size max(min_up_duration, min_down_duration) @@ -105,8 +105,11 @@ def find_min_generation_fast( Returns ------- - List[float] - Minimum generation power per timestep (MW), clamped by cluster_max_generation. + Tuple[List[float], List[float]] + Minimum and maximum generation power per timestep (MW), both clamped by + cluster_max_generation. If min_power_per_unit is ~0, the minimum list is all + zeros, but the maximum list still reflects the units kept committed by the + min_up/min_down window logic. """ min_up_duration = _round_to_int_duration(min_up_duration, "min_up_duration") min_down_duration = _round_to_int_duration(min_down_duration, "min_down_duration") @@ -116,9 +119,6 @@ def find_min_generation_fast( if isinstance(cluster_max_generation, (int, float)): cluster_max_generation = [float(cluster_max_generation)] * num_timesteps - if abs(min_power_per_unit) < TOL: - return [0.0] * num_timesteps - assert max_power_per_unit > TOL num_units_required = [math.ceil(p / max_power_per_unit) for p in generation_power] @@ -151,10 +151,16 @@ def find_min_generation_fast( num_units_on=num_units_on, ) - return [ - min(n * min_power_per_unit, cluster_max_generation[t]) - for t, n in enumerate(num_units_on) - ] + return ( + [ + min(n * min_power_per_unit, cluster_max_generation[t]) + for t, n in enumerate(num_units_on) + ], + [ + min(n * max_power_per_unit, cluster_max_generation[t]) + for t, n in enumerate(num_units_on) + ], + ) def find_num_units_accurate( diff --git a/tests/e2e/functional/libs/thermal_variants_for_heuristic.yml b/tests/e2e/functional/libs/thermal_variants_for_heuristic.yml index bd7011c2..a5778c6d 100644 --- a/tests/e2e/functional/libs/thermal_variants_for_heuristic.yml +++ b/tests/e2e/functional/libs/thermal_variants_for_heuristic.yml @@ -154,6 +154,6 @@ library: expression: expec(sum(market_bid_cost * generation_power)) extra-outputs: - id: num_units_on - expression: max(floor(lower_bound(generation_power) / min_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))) + expression: max(floor(upper_bound(generation_power) / max_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))) - id: non_prop_cost - expression: startup_cost * max(0,max(floor(lower_bound(generation_power) / min_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100)))-(max(floor(lower_bound(generation_power) / min_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))))[t-1])+ fixed_cost * max(floor(lower_bound(generation_power) / min_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))) + expression: startup_cost * max(0,max(floor(upper_bound(generation_power) / max_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100)))-(max(floor(upper_bound(generation_power) / max_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))))[t-1])+ fixed_cost * max(floor(upper_bound(generation_power) / max_power_per_unit), ceil(generation_power / max_power_per_unit / (1-spinning/100))) diff --git a/tests/e2e/functional/optim-config/thermal_variants_for_heuristic.yml b/tests/e2e/functional/optim-config/thermal_variants_for_heuristic.yml index d6154f80..625b5c6e 100644 --- a/tests/e2e/functional/optim-config/thermal_variants_for_heuristic.yml +++ b/tests/e2e/functional/optim-config/thermal_variants_for_heuristic.yml @@ -67,3 +67,6 @@ models: - heuristic-element: minimum_generation_power id: generation_power type: variable-lower-bound + - heuristic-element: maximum_generation_power + id: generation_power + type: variable-upper-bound diff --git a/tests/e2e/functional/studies/thermal_heuristic_one_cluster/input/data-series/demand-ts-only-hour13.txt b/tests/e2e/functional/studies/thermal_heuristic_one_cluster/input/data-series/demand-ts-only-hour13.txt new file mode 100644 index 00000000..cf8f7d00 --- /dev/null +++ b/tests/e2e/functional/studies/thermal_heuristic_one_cluster/input/data-series/demand-ts-only-hour13.txt @@ -0,0 +1,168 @@ +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +2050 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 +0 \ No newline at end of file diff --git a/tests/e2e/functional/test_thermal_heuristic_one_cluster_no_pmin.py b/tests/e2e/functional/test_thermal_heuristic_one_cluster_no_pmin.py new file mode 100644 index 00000000..2f2e8148 --- /dev/null +++ b/tests/e2e/functional/test_thermal_heuristic_one_cluster_no_pmin.py @@ -0,0 +1,181 @@ +# Copyright (c) 2026, RTE (https://www.rte-france.com) +# +# See AUTHORS.txt +# +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. +# +# SPDX-License-Identifier: MPL-2.0 +# +# This file is part of the Antares project. + +""" +End-to-end test for the thermal one-cluster study with P_min = 0 and a +single demand spike. + +Same study as tests/e2e/functional/test_thermal_heuristic_one_cluster.py +(one node, one thermal cluster G, one fixed demand D), except that: + - G's min_power_per_unit is overridden to 0 in memory (see + ``thermal_heuristic_helpers.with_parameter_value``). + - D's demand is overridden to 0 at every hour except the 13th (2050 MW), + using the dedicated data series + studies/thermal_heuristic_one_cluster/input/data-series/demand-ts-only-hour13.txt. + +With demand at 0 everywhere else, the cluster only ever needs to be +committed around the 13th-hour spike, and P_min = 0 means it is never +forced to produce anything while idling. min_up_duration = 3 still forces +whichever units start for the spike to stay committed for the 13th, 14th +and 15th hour (all 3 start together, so startup_cost applies to all 3). +""" + +import pytest + +from gems_runner.session.session import SimulationSession +from tests.e2e.functional.thermal_heuristic_helpers import ( + build_thermal_study, + check_output, + optim_config_for, +) + +CASE_ID = "one_cluster" +PARAMETER_OVERRIDES = { + "G": {"min_power_per_unit": 0}, + "D": {"load": "demand-ts-only-hour13"}, +} + + +def test_milp_version() -> None: + """ + The 2050 MW spike at the 13th hour needs all 3 units. Since they all start together, startup_cost applies to + all 3 (3 x 50). min_up_duration = 3 keeps them committed for the 13th, + 14th and 15th hour, but with P_min = 0 they produce nothing once + demand drops back to 0. + + The optimal cost is then : + 50 x 2050 (prod step 13) + + 3 x 1 (fixed cost step 13) + + 3 x 50 (start up step 13, all 3 units) + + 3 x 1 x 2 (fixed cost steps 14-15, still committed, P_min = 0 so no prod) + = 102 659 + """ + study = build_thermal_study(CASE_ID, "milp", PARAMETER_OVERRIDES) + config = optim_config_for("milp") + st = SimulationSession(study, config).run() + + assert st.data.loc[st.data["output"] == "objective-value", "value"].iloc[ + 0 + ] == pytest.approx(102659) + check_output( + st, + "G", + "non_prop_cost", + [153 if t == 12 else (3 if t in (13, 14) else 0) for t in range(168)], + ) + check_output( + st, "G", "generation_power", [2050 if t == 12 else 0 for t in range(168)] + ) + check_output( + st, "G", "num_units_on", [3 if t in (12, 13, 14) else 0 for t in range(168)] + ) + check_output(st, "N", "unsupplied_energy", [0.0] * 168) + check_output(st, "N", "spilled_energy", [0.0] * 168) + + +def test_lp_version() -> None: + """ + Same shape as the MILP solution but with a continuous unit count + (2.05, matching 2050 / 1000 MW), still held committed for 3 hours by + min_up_duration. + """ + study = build_thermal_study(CASE_ID, "lp", PARAMETER_OVERRIDES) + config = optim_config_for("lp") + st = SimulationSession(study, config).run() + + assert st.data.loc[st.data["output"] == "objective-value", "value"].iloc[ + 0 + ] == pytest.approx(102608.65) + check_output( + st, + "G", + "non_prop_cost", + [104.55 if t == 12 else (2.05 if t in (13, 14) else 0) for t in range(168)], + ) + check_output( + st, "G", "generation_power", [2050 if t == 12 else 0 for t in range(168)] + ) + check_output( + st, "G", "num_units_on", [2.05 if t in (12, 13, 14) else 0 for t in range(168)] + ) + check_output(st, "N", "unsupplied_energy", [0.0] * 168) + check_output(st, "N", "spilled_energy", [0.0] * 168) + + +def test_accurate_heuristic() -> None: + """ + Solve the same problem as before with the accurate heuristic. + + Ceiling the LP relaxation's 2.05 units at the 13th hour gives 3, which + is already the MILP-optimal number of units, so the accurate heuristic + retrieves the exact MILP solution. + """ + study = build_thermal_study(CASE_ID, "accurate", PARAMETER_OVERRIDES) + config = optim_config_for("accurate") + st = SimulationSession(study, config).run() + + assert st.data.loc[st.data["output"] == "objective-value", "value"].iloc[ + 0 + ] == pytest.approx(102659) + check_output( + st, + "G", + "non_prop_cost", + [153 if t == 12 else (3 if t in (13, 14) else 0) for t in range(168)], + ) + check_output( + st, "G", "generation_power", [2050 if t == 12 else 0 for t in range(168)] + ) + check_output( + st, "G", "num_units_on", [3 if t in (12, 13, 14) else 0 for t in range(168)] + ) + check_output(st, "N", "unsupplied_energy", [0.0] * 168) + check_output(st, "N", "spilled_energy", [0.0] * 168) + + +def test_fast_heuristic() -> None: + """ + Solve the same problem as before with the fast heuristic. + + The fast heuristic derives maximum_generation_power from a sliding window of + size max(min_up_duration, min_down_duration) = max(3, 10) = 10 hours, so the 3 + committed units stay available for the whole 10-hour window containing the + 13th-hour spike (hours 11 to 20, 1-indexed), not just for min_up_duration. As + with every other fast-heuristic test in this module, the reported objective + only reflects the proportional generation cost, not the fixed/startup costs + (``non_prop_cost``). + + The optimal cost is then : + 50 x 2050 (prod step 13) + = 102 500 + """ + study = build_thermal_study(CASE_ID, "fast", PARAMETER_OVERRIDES) + config = optim_config_for("fast") + st = SimulationSession(study, config).run() + + assert st.data.loc[st.data["output"] == "objective-value", "value"].iloc[ + 0 + ] == pytest.approx(102500) + check_output( + st, + "G", + "non_prop_cost", + [153 if t == 10 else (3 if 11 <= t <= 19 else 0) for t in range(168)], + ) + check_output( + st, "G", "generation_power", [2050 if t == 12 else 0 for t in range(168)] + ) + check_output( + st, "G", "num_units_on", [3 if 10 <= t <= 19 else 0 for t in range(168)] + ) + check_output(st, "N", "unsupplied_energy", [0.0] * 168) + check_output(st, "N", "spilled_energy", [0.0] * 168) diff --git a/tests/e2e/functional/thermal_heuristic_helpers.py b/tests/e2e/functional/thermal_heuristic_helpers.py index c3958be7..4827237a 100644 --- a/tests/e2e/functional/thermal_heuristic_helpers.py +++ b/tests/e2e/functional/thermal_heuristic_helpers.py @@ -35,7 +35,7 @@ from dataclasses import dataclass from functools import lru_cache from pathlib import Path -from typing import Any, Dict, List, cast +from typing import Any, Dict, List, Optional, Union, cast import pandas as pd import pytest @@ -106,6 +106,19 @@ def with_integer_strategy( return new_system +def with_parameter_value( + system: SystemSchema, component_id: str, parameter_id: str, value: Union[float, str] +) -> SystemSchema: + """Return a deep copy of *system* with one component's parameter value overridden.""" + new_system = system.model_copy(deep=True) + for comp in new_system.components: + if comp.id == component_id: + for param in comp.parameters: + if param.id == parameter_id: + param.value = value + return new_system + + # --------------------------------------------------------------------------- # Case registry # --------------------------------------------------------------------------- @@ -148,11 +161,19 @@ def _shared_library() -> LibrarySchema: return parse_yaml_library(f) -def build_thermal_study(case_id: str, mode: str) -> Study: +def build_thermal_study( + case_id: str, + mode: str, + parameter_overrides: Optional[Dict[str, Dict[str, Union[float, str]]]] = None, +) -> Study: """Build a Study in memory for (case_id, mode) in {milp, lp, accurate, fast}. No disk writes: every variant is derived from schemas parsed once from the case's committed base directory and the shared library file. + + ``parameter_overrides`` optionally maps component id -> {parameter id: value}, + applied on top of the base study's parameters (e.g. to zero out a thermal + cluster's P_min without committing a dedicated study directory). """ spec = CASES[case_id] system = _base_system(spec.base_dir) @@ -161,6 +182,11 @@ def build_thermal_study(case_id: str, mode: str) -> Study: if mode != "milp": system = with_integer_strategy(system, spec.thermal_components, mode) + if parameter_overrides: + for component_id, params in parameter_overrides.items(): + for parameter_id, value in params.items(): + system = with_parameter_value(system, component_id, parameter_id, value) + resolved_system = resolve_system(system, lib_dict) series_dir = STUDIES_DIR / spec.base_dir / "input" / "data-series" diff --git a/tests/unittests/gems_craft/optim_config/test_heuristic_validation.py b/tests/unittests/gems_craft/optim_config/test_heuristic_validation.py index 4287564f..cd9dc44d 100644 --- a/tests/unittests/gems_craft/optim_config/test_heuristic_validation.py +++ b/tests/unittests/gems_craft/optim_config/test_heuristic_validation.py @@ -106,6 +106,11 @@ def _fast_heuristic_config( id="generation_power", type=ModelElementAccessType.VARIABLE_LOWER_BOUND, ), + HeuristicElementConfig( + heuristic_element="maximum_generation_power", + id="generation_power", + type=ModelElementAccessType.VARIABLE_UPPER_BOUND, + ), ], ) diff --git a/tests/unittests/gems_runner/simulation/test_thermal_heuristic.py b/tests/unittests/gems_runner/simulation/test_thermal_heuristic.py index d892aacd..9bea1559 100644 --- a/tests/unittests/gems_runner/simulation/test_thermal_heuristic.py +++ b/tests/unittests/gems_runner/simulation/test_thermal_heuristic.py @@ -36,8 +36,24 @@ def test_find_min_generation_fast_scalar_matches_constant_list() -> None: def test_find_min_generation_fast_scalar_clamps_output() -> None: - result = find_min_generation_fast(_GENERATION_POWER, 40.0, 50.0, 50.0, 2, 2) - assert all(v <= 40.0 for v in result) + minimum, maximum = find_min_generation_fast( + _GENERATION_POWER, 40.0, 50.0, 50.0, 2, 2 + ) + assert all(v <= 40.0 for v in minimum) + assert all(v <= 40.0 for v in maximum) + + +def test_find_min_generation_fast_no_pmin_keeps_units_committed() -> None: + """With min_power_per_unit = 0, the minimum output is all zeros, but the maximum + output must still reflect units kept committed by the min_up/min_down window logic + (no early-return short-circuit).""" + generation_power = [0.0] * 5 + [100.0] + [0.0] * 5 + minimum, maximum = find_min_generation_fast( + generation_power, 200.0, 0.0, 50.0, 3, 3 + ) + assert minimum == [0.0] * len(generation_power) + assert maximum[5] == pytest.approx(100.0) + assert any(v > 0.0 for t, v in enumerate(maximum) if t != 5) # --------------------------------------------------------------------------- From c9e2998681a7478e380117657537a538ec9c3995 Mon Sep 17 00:00:00 2001 From: "Antoine Oustry, PhD" <58943406+aoustry@users.noreply.github.com> Date: Mon, 24 Aug 2026 11:18:12 +0200 Subject: [PATCH 3/4] sequential mode: configurable carry-over-length --- docs/CHANGELOG.md | 2 + docs/user-guide/optim-config.md | 92 ++++++- src/gems_craft/optim_config/parsing.py | 64 +++++ src/gems_runner/session/session.py | 37 ++- src/gems_runner/simulation/optimization.py | 60 ++++- .../test_sequential_carry_over_length.py | 247 ++++++++++++++++++ .../optim_config/test_resolution_config.py | 229 ++++++++++++++++ .../test_carry_over_initial_values.py | 111 ++++++++ 8 files changed, 815 insertions(+), 27 deletions(-) create mode 100644 tests/e2e/functional/test_sequential_carry_over_length.py create mode 100644 tests/unittests/gems_craft/optim_config/test_resolution_config.py create mode 100644 tests/unittests/gems_runner/simulation/test_carry_over_initial_values.py diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 63af7d27..145c0872 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -5,6 +5,8 @@ All notable changes to GemsPy are documented here. ## [Unreleased] ### Changed +- **Sequential mode carry-over length can now be controlled by the user through the parameter `carry-over-length` (default: `block-overlap`). This also fixes an incorrect stitching for `block-overlap >= 2`, where the previous hardcoded behaviour pinned block + *N+1*'s first timestep to block *N*'s **last** timestep — a different absolute timestep. - **linopy upgraded to `>=0.9.0`** - the minimum supported Python version rises to **3.11** accordingly (linopy 0.9 requires Python >= 3.11). diff --git a/docs/user-guide/optim-config.md b/docs/user-guide/optim-config.md index 0111c6c4..55f7fc82 100644 --- a/docs/user-guide/optim-config.md +++ b/docs/user-guide/optim-config.md @@ -41,7 +41,8 @@ solver-options: resolution: mode: sequential-subproblems # see section below block-length: 168 # one week (in timesteps) - block-overlap: 0 + block-overlap: 24 # consecutive blocks share one day + carry-over-length: 24 # optional; omitted → defaults to block-overlap # Per-model configuration (optional) models: @@ -220,7 +221,8 @@ optimisation subproblems. |---|---|---|---| | `mode` | str | `"frontal"` | Resolution strategy (see below) | | `block-length` | int | — | Timesteps per window; required for windowed modes | -| `block-overlap` | int | `0` | Extra overlap timesteps between consecutive blocks | +| `block-overlap` | int | `0` | Sequential mode only (rejected in other modes): shared timesteps between consecutive blocks; must satisfy `0 <= block-overlap < block-length` | +| `carry-over-length` | int | `block-overlap` | Sequential mode only (rejected in other modes): how many of the shared timesteps are pinned to the previous block's values; must satisfy `0 <= carry-over-length <= block-overlap` | ### `frontal` (default) @@ -236,18 +238,92 @@ Produces globally optimal results. ### `sequential-subproblems` -The horizon is split into non-overlapping (or slightly overlapping) windows of -`block-length` timesteps. Blocks are solved **one after the other**; the state -of inter-block dynamics (e.g. storage level) is carried over from one block to -the next. +The horizon is split into windows of `block-length` timesteps, each starting +`block-length - block-overlap` timesteps after the previous one. Blocks are +solved **one after the other**; the state of inter-block dynamics (e.g. storage +level) is carried over from one block to the next by pinning the leading +timesteps of each block to the values the previous block already computed. ~~~ yaml resolution: mode: sequential-subproblems - block-length: 168 # one week - block-overlap: 0 + block-length: 168 # one week + block-overlap: 24 # one day shared between consecutive blocks + carry-over-length: 24 # optional; omitted → defaults to block-overlap (full pin) + # 0 is legal and explicit: overlap solved twice, no stitching ~~~ +Three parameters shape the stitching between consecutive blocks. Illustrative +example with `block-length: 10`, `block-overlap: 4`, `carry-over-length: 3` +(a partial pin, so all three parameters are visible at once): + +~~~ text +abs t 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +Block N 0 1 2 3 4 5 6 7 8 9 + └──────────────────────────────────────┘ + block-length = 10 + +Block N+1 0 1 2 3 4 5 6 7 8 9 + └──────────────────────────────────────┘ + block-length = 10 + + |------------| overlap = 4 (t=6..9: solved by BOTH blocks) + |========| carry-over = 3 (t=6..8: PINNED to block N's value) + ^ t=9: still shared, but free in N+1 (re-optimized) +~~~ + +Reading it: + +- **`block-length`** — width of each block's own window (10 for both here). +- **`block-overlap`** — how far block *N+1*'s start reaches back into block + *N*'s window (4 → t=6..9 exist in both solves). The overlap gives block + *N+1* real historical values for lag-dependent constraints (e.g. a storage + balance using `soc[t-1]`, or min up/down durations spanning several hours). +- **`carry-over-length`** — how many of those *shared* leading timesteps of + block *N+1* get hard-pinned (`var[t] == value from block N`) to block *N*'s + already-solved values, counted from the earliest shared timestep (t=6), not + from t=9. Here `carry-over-length: 3 < overlap: 4`, so t=6,7,8 are frozen + but t=9 is left free — an MPC-style partial pin where the optimizer may + revise the tail of the overlap with more lookback context. + +Defaults and special values: + +- **Omitted** `carry-over-length` resolves to `block-overlap`: the whole + overlap zone is pinned. This is the right default when the overlap exists + to provide history for lag-dependent constraints without re-litigating + decisions the previous block already made. +- **Explicit `carry-over-length: 0`** is legal and distinct from omitting the + field: blocks overlap for lag-constraint history, but no timestep is pinned + — block *N+1* re-solves the whole overlap window independently. +- Validation requires `0 <= carry-over-length <= block-overlap` (and + `0 <= block-overlap < block-length`), with no special case at + `block-overlap: 0`. + +Overlapping timesteps appear once per block in the simulation table, tagged +with the `block` column — nothing is lost or silently merged. Downstream +tooling decides which block's version of a shared timestep is authoritative; +`carry-over-length` only controls how much two consecutive blocks may +*disagree* on that shared window. + +**What the carry-over pins.** The mechanism is plain *variable fixing*: for +block *N+1*, every time-dependent variable whose block-relative timestep falls +in `[0, carry-over-length[` is fixed to the value block *N* computed for the +**same absolute timestep**. Two consequences are worth spelling out: + +- It is **not** an initial-condition mechanism. Block *N+1*'s problem is not + given the value of the timestep *preceding* its window, so a `t-1` time-shift + operator at the block's first timestep still resolves against that block's own + border condition (cyclic by default) rather than reaching into block *N*. +- It applies to **all** time-dependent variables of all models, not only + state-like ones such as a storage level. Finer, per-model granularity can be + added later if a use case needs it. + +**Time-independent** variables (`structure.time = False`, e.g. an investment +capacity) are never carried over — nothing links their values across blocks, so +each block sizes them independently. Sequential mode is therefore not suited to +investment problems; use `frontal` or `benders-decomposition` for those. + + ### `parallel-subproblems` diff --git a/src/gems_craft/optim_config/parsing.py b/src/gems_craft/optim_config/parsing.py index fcf6ae40..e4a19e77 100644 --- a/src/gems_craft/optim_config/parsing.py +++ b/src/gems_craft/optim_config/parsing.py @@ -149,10 +149,14 @@ class ResolutionMode(str, Enum): BENDERS_DECOMPOSITION = "benders-decomposition" +_SEQUENTIAL_ONLY_FIELDS = ("block_overlap", "carry_over_length") + + class ResolutionConfig(ModifiedBaseModel): mode: ResolutionMode = ResolutionMode.FRONTAL block_length: Optional[int] = None block_overlap: int = 0 + carry_over_length: Optional[int] = None @model_validator(mode="after") def _block_length_required_for_windowed_modes(self) -> "ResolutionConfig": @@ -164,6 +168,66 @@ def _block_length_required_for_windowed_modes(self) -> "ResolutionConfig": raise ValueError(f"'block_length' is required for mode '{self.mode.value}'") return self + @model_validator(mode="after") + def _reject_sequential_only_fields(self) -> "ResolutionConfig": + """'block-overlap' and 'carry-over-length' steer the stitching of + consecutive blocks, which only exists in sequential mode. Reject them + elsewhere instead of dropping them silently. The check is on the keys + the user actually wrote (``model_fields_set``), so an explicit + 'block-overlap: 0' is rejected too.""" + if self.mode == ResolutionMode.SEQUENTIAL_SUBPROBLEMS: + return self + declared = [ + name for name in _SEQUENTIAL_ONLY_FIELDS if name in self.model_fields_set + ] + if declared: + keys = ", ".join(f"'{name.replace('_', '-')}'" for name in declared) + plural = len(declared) > 1 + raise ValueError( + f"{keys} only appl{'y' if plural else 'ies'} to mode " + f"'{ResolutionMode.SEQUENTIAL_SUBPROBLEMS.value}', but mode is " + f"'{self.mode.value}'; remove {'them' if plural else 'it'} " + f"or switch mode" + ) + return self + + @model_validator(mode="after") + def _validate_block_overlap(self) -> "ResolutionConfig": + if self.block_overlap < 0: + raise ValueError(f"'block-overlap' must be >= 0, got {self.block_overlap}") + if self.block_length is not None and self.block_overlap >= self.block_length: + raise ValueError( + f"'block-overlap' ({self.block_overlap}) must be < 'block-length' " + f"({self.block_length})" + ) + return self + + @model_validator(mode="after") + def _validate_carry_over_length(self) -> "ResolutionConfig": + if self.carry_over_length is not None: + if self.carry_over_length < 0: + raise ValueError( + f"'carry-over-length' must be >= 0, got {self.carry_over_length}" + ) + if self.carry_over_length > self.block_overlap: + raise ValueError( + f"'carry-over-length' ({self.carry_over_length}) must be <= " + f"'block-overlap' ({self.block_overlap})" + ) + return self + + @property + def effective_carry_over_length(self) -> int: + """Resolved carry-over length: explicit value if set, else full pin of + the overlap zone (``block_overlap``). ``0`` is a legal explicit value, + distinct from "unset", meaning blocks overlap for lag-constraint + history but are not stitched at all.""" + return ( + self.carry_over_length + if self.carry_over_length is not None + else self.block_overlap + ) + class TimeScopeConfig(ModifiedBaseModel): first_time_step: int = 0 diff --git a/src/gems_runner/session/session.py b/src/gems_runner/session/session.py index 9dc14849..66e56918 100644 --- a/src/gems_runner/session/session.py +++ b/src/gems_runner/session/session.py @@ -91,6 +91,7 @@ def _run_sequential(self) -> SimulationTable: cfg = self.optim_config.resolution block_length: int = cfg.block_length # type: ignore[assignment] block_overlap: int = cfg.block_overlap + carry_over_length: int = cfg.effective_carry_over_length tables: List[SimulationTable] = [] for scenario_id in self.scenario_ids: @@ -111,10 +112,16 @@ def _run_sequential(self) -> SimulationTable: initial_values=carry_over or None, ) tables.append(table) + # Block N and block N+1 share `block_overlap` absolute + # timesteps: block N's local indices `block_length - overlap + # ...` are block N+1's local indices `0 ...`. + delta = block_length - block_overlap + t_start += delta carry_over = self._extract_carry_over( - problem, local_index=len(timesteps) - 1 + problem, + local_start=delta, + length=carry_over_length, ) - t_start += block_length - block_overlap block_id += 1 return self._reduce(tables) @@ -240,17 +247,33 @@ def _reduce(self, tables: List[SimulationTable]) -> SimulationTable: @staticmethod def _extract_carry_over( problem: OptimizationProblem, - local_index: int, + local_start: int, + length: int, ) -> Dict[Tuple[str, str], xr.DataArray]: - """Extract variable values at *local_index* for use as initial values in the next block.""" + """Extract variable values over *length* timesteps starting at *local_start*. + + The returned arrays keep a ``time`` dimension re-indexed to + ``0 .. length-1`` so they align with the leading timesteps of the next + block's variables. The window is clamped to the solved block's actual + horizon (a truncated final block can be shorter than ``block_length``), + so fewer than *length* values may be carried over. + + Only variables carrying a ``time`` dimension are extracted. + Time-independent variables (e.g. an investment capacity) are + deliberately left free in every block, so each block re-optimizes them + independently. + """ carry_over: Dict[Tuple[str, str], xr.DataArray] = {} - if problem.linopy_model.solution is None: + if length <= 0 or problem.linopy_model.solution is None: return carry_over for (model, var_name), linopy_var in problem._linopy_vars.items(): if "time" in linopy_var.dims: sol_da = problem.get_variable_solution(model, var_name) if sol_da is not None: - carry_over[(model, var_name)] = sol_da.isel( - time=local_index, drop=True + window = sol_da.isel(time=slice(local_start, local_start + length)) + if window.sizes["time"] == 0: + continue + carry_over[(model, var_name)] = window.assign_coords( + time=list(range(window.sizes["time"])) ) return carry_over diff --git a/src/gems_runner/simulation/optimization.py b/src/gems_runner/simulation/optimization.py index cc1905ba..71752c58 100644 --- a/src/gems_runner/simulation/optimization.py +++ b/src/gems_runner/simulation/optimization.py @@ -506,13 +506,35 @@ def get_variable_upper_bound( # --------------------------------------------------------------------------- +def _validate_initial_values( + initial_values: Optional[Dict[Tuple[str, str], xr.DataArray]], +) -> Dict[Tuple[str, str], xr.DataArray]: + """Check the carry-over contract on *initial_values* and return them. + + Every value must carry a ``time`` dimension indexed ``0 .. k-1`` so that it + aligns with the leading timesteps of the block being built. + """ + values = initial_values or {} + for (mk, var_name), init_val in values.items(): + if "time" not in init_val.dims: + raise ValueError( + f"initial_values[{mk!r}, {var_name!r}] must carry a 'time' " + f"dimension indexed 0..k-1; got dims {tuple(init_val.dims)}" + ) + return values + + class _OptimizationProblemBuilder: """ - Builds the linopy problem in 4 phases: + Builds the linopy problem in 5 phases: 1. Build parameter DataArrays for all models. 2. Create all linopy Variables (uses param arrays for bounds). 3. Build port arrays via incidence matrices. 4. Add constraints and objectives to the linopy model. + 5. Add the carry-over constraints of *initial_values* (sequential mode): + each time-dependent variable is *fixed*, over the block's first ``k`` + timesteps, to the value the previous block computed for the same + absolute timestep. Time-independent variables are never pinned. """ def __init__( @@ -531,7 +553,7 @@ def __init__( self.scenario_ids = scenario_ids self._location_filter = location_filter self._oob_filter = oob_filter - self._initial_values = initial_values or {} + self._initial_values = _validate_initial_values(initial_values) self.block_length = len(block.timesteps) self.time_coord = list(range(self.block_length)) @@ -573,15 +595,24 @@ def build(self) -> OptimizationProblem: model, port_arrays_for_model, total_obj ) - # Phase 5: carry-over constraints (sequential mode only) + # Phase 5: carry-over constraints (sequential mode only). + # Only time-dependent variables are pinned: time-independent ones + # (structure.time = False, e.g. an investment capacity) are deliberately + # left free in every block — see the user guide, `sequential-subproblems`. for (mk, var_name), init_val in self._initial_values.items(): linopy_var = self.linopy_vars.get((mk, var_name)) - if linopy_var is not None and "time" in linopy_var.dims: - safe = f"{mk}__{var_name}".replace("-", "_") - self.linopy_model.add_constraints( - linopy_var.isel(time=0) == init_val, # type: ignore[arg-type] - name=f"carry_over__{safe}", - ) + if linopy_var is None or "time" not in linopy_var.dims: + continue + # Pin the first len(init_val.time) timesteps, clamped to this + # block's horizon (a truncated final block can be shorter than the + # carried window). + pin_length = min(init_val.sizes["time"], self.block_length) + safe = f"{mk}__{var_name}".replace("-", "_") + self.linopy_model.add_constraints( + linopy_var.isel(time=slice(0, pin_length)) + == init_val.isel(time=slice(0, pin_length)), # type: ignore[arg-type] + name=f"carry_over__{safe}", + ) # Extract constant objective contribution (linopy cannot hold pure constants). objective_constant = 0.0 @@ -1031,9 +1062,14 @@ def build_problem( problem_name: Label for the linopy model. initial_values: - Optional carry-over values keyed by ``(model_id, var_name)``. For - each entry a constraint ``var[time=0] == value`` is added, overriding - the cyclic border condition for the first timestep. + Optional carry-over values keyed by ``(model_id, var_name)``. Each + value must be an ``xr.DataArray`` carrying a ``time`` dimension of + length ``k`` indexed ``0 .. k-1``; constraints + ``var[time=i] == value[i]`` are then added for the block's first ``k`` + timesteps, overriding the cyclic border condition on that window. A + value without a ``time`` dimension raises ``ValueError`` before the + problem is built. Entries whose variable is time-independent, or is + absent from this block, are ignored. """ study.check_consistency() diff --git a/tests/e2e/functional/test_sequential_carry_over_length.py b/tests/e2e/functional/test_sequential_carry_over_length.py new file mode 100644 index 00000000..7c28af7c --- /dev/null +++ b/tests/e2e/functional/test_sequential_carry_over_length.py @@ -0,0 +1,247 @@ +# Copyright (c) 2024, RTE (https://www.rte-france.com) +# +# See AUTHORS.txt +# +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. +# +# SPDX-License-Identifier: MPL-2.0 +# +# This file is part of the Antares project. + +""" +E2E test: multi-timestep carry-over in sequential mode. + +Reuses the rolling_horizon_suboptimality study (generator p_max=2/cost=1, +storage capacity=2/rate=2, bus with ens_cost=100) with a longer, aperiodic +12-step demand series, supplied in memory rather than through a study folder: +the study directory and its `optim-config.yml` are only an entry point, and +`run_study`'s folder-to-CSV path has its own test (test_study_from_folder.py). + +Sequential mode with block-length=6, block-overlap=3 over t=0..11: + + block 0: [0 1 2 3 4 5] + block 1: [3 4 5 6 7 8] + block 2: [6 7 8 9 10 11] + block 3: [9 10 11] (truncated tail) + +Consecutive blocks share `block-overlap` = 3 absolute timesteps. The +`carry-over-length` first shared timesteps of block N+1 are pinned to block +N's already-solved values — counted from the *earliest* shared timestep, so +each pinned constraint matches the same absolute timestep in both blocks. + +`carry-over-length` is checked on the carry-over constraints themselves: what +the setting controls is which variables are *fixed* and which are left free, +and comparing solved values cannot tell a free timestep that happens to +re-optimize to the same value from a fixed one. +""" + +from functools import lru_cache +from pathlib import Path +from typing import Any, Dict, List, Set, Tuple + +import pandas as pd +import pytest + +from gems_craft.optim_config.parsing import ( + ModelOptimConfig, + OptimConfig, + OutOfBoundsConstraintConfig, + OutOfBoundsMode, + OutOfBoundsProcessingConfig, + ResolutionConfig, + ResolutionMode, + ScenarioScopeConfig, + TimeScopeConfig, +) +from gems_craft.study.data import TimeSeriesData +from gems_craft.study.folder import load_study +from gems_craft.study.study import Study +from gems_runner.session.session import SimulationSession +from gems_runner.simulation.optimization import OptimizationProblem +from gems_runner.simulation.simulation_table import SimulationTable + +_STUDY_SRC = Path(__file__).parent / "studies" / "rolling_horizon_suboptimality" + +# Aperiodic demand: peaks (4) need gen=2 + discharge=2, mid steps (2) are +# covered by the generator alone, zeros allow recharging. The pattern is +# deliberately not periodic with the block stride so that the SoC trajectory +# differs between a block's last timestep and the start of the overlap zone. +_DEMAND = [0, 4, 2, 0, 4, 0, 2, 4, 0, 4, 4, 0] +_BLOCK_LENGTH = 6 +_BLOCK_OVERLAP = 3 + + +@lru_cache +def _study() -> Study: + """The committed rolling-horizon study, with its 6-step demand series + replaced in memory by the 12-step aperiodic one above.""" + study = load_study(_STUDY_SRC) + study.database.add_data( + "load_node", "demand", TimeSeriesData(pd.Series(_DEMAND, dtype=float)) + ) + return study + + +def _config(**resolution: Any) -> OptimConfig: + """The study's optim-config, with `resolution` built from the given fields. + + Fields left out are left *unset* (not defaulted), which is what + distinguishes an omitted `carry-over-length` from an explicit `0`, and what + keeps `block-overlap` — sequential-only — out of the parallel config. + """ + return OptimConfig( + time_scope=TimeScopeConfig(first_time_step=0, last_time_step=len(_DEMAND) - 1), + scenario_scope=ScenarioScopeConfig(include=[0]), + models=[ + ModelOptimConfig( + id="rolling-horizon-lib.storage", + out_of_bounds_processing=OutOfBoundsProcessingConfig( + constraints=[ + OutOfBoundsConstraintConfig( + id="soc_balance", mode=OutOfBoundsMode.DROP + ) + ] + ), + ) + ], + resolution=ResolutionConfig(**resolution), + ) + + +def _solve(config: OptimConfig) -> Tuple[SimulationTable, List[OptimizationProblem]]: + """Run the study through a `SimulationSession` and return its result table + plus the solved problems, one per block, in solve order. + + `SimulationSession._run_block` returns the solved problem for carry-over + extraction *or inspection*, which is what gives the test access to the + carry-over constraints. + """ + session = SimulationSession(_study(), config) + problems: List[OptimizationProblem] = [] + run_block = session._run_block + + def spy(*args: Any, **kwargs: Any) -> Any: + problem, table = run_block(*args, **kwargs) + problems.append(problem) + return problem, table + + session._run_block = spy # type: ignore[assignment] + return session.run(), problems + + +def _pinned_window_lengths(problem: OptimizationProblem) -> Set[int]: + """Number of timesteps each carry-over equality constraint of `problem` + fixes. Empty when the block carries nothing over.""" + linopy_model = problem.linopy_model + return { + int(linopy_model.constraints[name].sizes["time"]) + for name in linopy_model.constraints + if name.startswith("carry_over__") + } + + +def _value( + st: SimulationTable, block: int, component: str, output: str, timestep: int +) -> float: + df = st.data + rows = df[ + (df["block"] == block) + & (df["component"] == component) + & (df["output"] == output) + & (df["absolute_time_index"] == timestep) + ] + assert len(rows) == 1, ( + f"Expected exactly one row for block={block} component={component} " + f"output={output} t={timestep}, got {len(rows)}" + ) + return float(rows.iloc[0]["value"]) + + +@pytest.mark.parametrize( + "carry_over, expected", + [ + # Omitted resolves to `block-overlap`: the whole overlap zone is pinned. + ({}, _BLOCK_OVERLAP), + ({"carry_over_length": 0}, 0), + ({"carry_over_length": 1}, 1), + ({"carry_over_length": 2}, 2), + ], +) +def test_carry_over_length_fixes_that_many_leading_timesteps( + carry_over: Dict[str, int], expected: int +) -> None: + """`carry-over-length: k` fixes, in every block but the first, the k + leading local timesteps — the k earliest shared timesteps — to the previous + block's solution, and leaves the rest of the overlap zone free. + + `k = 0` fixes nothing at all: the blocks still overlap (so lag constraints + keep their history) but are not stitched. + """ + _, problems = _solve( + _config( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=_BLOCK_LENGTH, + block_overlap=_BLOCK_OVERLAP, + **carry_over, + ) + ) + # block-length=6, block-overlap=3, t=0..11 → blocks [0..5], [3..8], [6..11] + # and the truncated tail [9..11]. + assert len(problems) == 4 + assert not _pinned_window_lengths( + problems[0] + ), "Nothing is carried into the first block" + + for block_id, problem in enumerate(problems[1:], start=1): + windows = _pinned_window_lengths(problem) + assert windows == ({expected} if expected else set()), ( + f"Block {block_id}: every carry-over constraint must fix the " + f"{expected} leading timesteps, found {sorted(windows)}" + ) + + +def test_zero_overlap_blocks_fully_independent() -> None: + """With block-overlap: 0 nothing is carried between blocks: each block is + solved as if it were alone (no carry-over constraints). + + Two complementary checks: + + - Block 1 ([6..11], demand [2,4,0,4,4,0]) serves its t=7 peak by + pre-charging its *free* initial storage state. + - The whole solution is identical to parallel-subproblems mode, which + solves the same windows independently by construction (and where + `block-overlap` is not accepted at all). + """ + seq, _ = _solve( + _config( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=_BLOCK_LENGTH, + block_overlap=0, + ) + ) + par, _ = _solve( + _config(mode=ResolutionMode.PARALLEL_SUBPROBLEMS, block_length=_BLOCK_LENGTH) + ) + + assert _value(seq, 1, "bus", "unsupplied", 7) == pytest.approx( + 0.0, abs=1e-6 + ), "Block 1 must serve its t=7 peak from a free initial storage state" + + # Both modes enumerate the same windows with the same 0-based block ids. + for component, output in [ + ("storage", "soc"), + ("storage", "charge"), + ("storage", "discharge"), + ("gen", "p"), + ("bus", "unsupplied"), + ]: + for t in range(len(_DEMAND)): + block = t // _BLOCK_LENGTH + v_seq = _value(seq, block, component, output, t) + v_par = _value(par, block, component, output, t) + assert v_seq == pytest.approx(v_par, abs=1e-6), ( + f"sequential (block-overlap: 0) and parallel modes disagree at " + f"t={t} for {component}.{output}: {v_seq} != {v_par}" + ) diff --git a/tests/unittests/gems_craft/optim_config/test_resolution_config.py b/tests/unittests/gems_craft/optim_config/test_resolution_config.py new file mode 100644 index 00000000..2cd01598 --- /dev/null +++ b/tests/unittests/gems_craft/optim_config/test_resolution_config.py @@ -0,0 +1,229 @@ +# Copyright (c) 2024, RTE (https://www.rte-france.com) +# +# See AUTHORS.txt +# +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. +# +# SPDX-License-Identifier: MPL-2.0 +# +# This file is part of the Antares project. + +import pytest +from pydantic import ValidationError + +from gems_craft.optim_config.parsing import ResolutionConfig, ResolutionMode + +# --------------------------------------------------------------------------- +# Defaults and parsing +# --------------------------------------------------------------------------- + + +def test_defaults() -> None: + cfg = ResolutionConfig() + assert cfg.mode == ResolutionMode.FRONTAL + assert cfg.block_length is None + assert cfg.block_overlap == 0 + assert cfg.carry_over_length is None + assert cfg.effective_carry_over_length == 0 + + +def test_kebab_case_aliases() -> None: + cfg = ResolutionConfig.model_validate( + { + "mode": "sequential-subproblems", + "block-length": 168, + "block-overlap": 24, + "carry-over-length": 12, + } + ) + assert cfg.block_length == 168 + assert cfg.block_overlap == 24 + assert cfg.carry_over_length == 12 + + +def test_block_length_required_for_windowed_modes() -> None: + with pytest.raises(ValidationError, match="block_length"): + ResolutionConfig(mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS) + + +# --------------------------------------------------------------------------- +# effective_carry_over_length resolution +# --------------------------------------------------------------------------- + + +def test_carry_over_defaults_to_block_overlap() -> None: + cfg = ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=168, + block_overlap=24, + ) + assert cfg.carry_over_length is None + assert cfg.effective_carry_over_length == 24 + + +def test_explicit_zero_is_distinct_from_unset() -> None: + cfg = ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=168, + block_overlap=24, + carry_over_length=0, + ) + assert cfg.carry_over_length == 0 + assert cfg.effective_carry_over_length == 0 + + +def test_partial_carry_over() -> None: + cfg = ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=4, + carry_over_length=3, + ) + assert cfg.effective_carry_over_length == 3 + + +# --------------------------------------------------------------------------- +# block-overlap validation +# --------------------------------------------------------------------------- + + +def test_negative_block_overlap_rejected() -> None: + with pytest.raises(ValidationError, match="'block-overlap' must be >= 0"): + ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=-1, + ) + + +def test_block_overlap_equal_to_block_length_rejected() -> None: + with pytest.raises(ValidationError, match="must be < 'block-length'"): + ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=10, + ) + + +def test_block_overlap_greater_than_block_length_rejected() -> None: + with pytest.raises(ValidationError, match="must be < 'block-length'"): + ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=11, + ) + + +# --------------------------------------------------------------------------- +# carry-over-length validation +# --------------------------------------------------------------------------- + + +def test_negative_carry_over_length_rejected() -> None: + with pytest.raises(ValidationError, match="'carry-over-length' must be >= 0"): + ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=2, + carry_over_length=-1, + ) + + +def test_carry_over_length_greater_than_overlap_rejected() -> None: + with pytest.raises(ValidationError, match="must be <= 'block-overlap'"): + ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=2, + carry_over_length=3, + ) + + +def test_carry_over_length_rejected_when_overlap_is_zero() -> None: + # No special case at block_overlap == 0: any positive carry-over-length + # is rejected, there is no implicit single-timestep seeding any more. + with pytest.raises(ValidationError, match="must be <= 'block-overlap'"): + ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=0, + carry_over_length=1, + ) + + +def test_carry_over_length_equal_to_overlap_accepted() -> None: + cfg = ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=4, + carry_over_length=4, + ) + assert cfg.effective_carry_over_length == 4 + + +# --------------------------------------------------------------------------- +# Sequential-only fields rejected in other modes +# --------------------------------------------------------------------------- + +_NON_SEQUENTIAL_MODES = [ + ResolutionMode.FRONTAL, + ResolutionMode.PARALLEL_SUBPROBLEMS, + ResolutionMode.BENDERS_DECOMPOSITION, +] + + +@pytest.mark.parametrize("mode", _NON_SEQUENTIAL_MODES) +def test_block_overlap_rejected_outside_sequential(mode: ResolutionMode) -> None: + with pytest.raises(ValidationError, match="'block-overlap' only applies to mode"): + ResolutionConfig(mode=mode, block_length=10, block_overlap=2) + + +@pytest.mark.parametrize("mode", _NON_SEQUENTIAL_MODES) +def test_carry_over_length_rejected_outside_sequential(mode: ResolutionMode) -> None: + with pytest.raises( + ValidationError, match="'carry-over-length' only applies to mode" + ): + ResolutionConfig(mode=mode, block_length=10, carry_over_length=1) + + +def test_both_sequential_only_fields_reported_together() -> None: + with pytest.raises( + ValidationError, + match="'block-overlap', 'carry-over-length' only apply to mode", + ): + ResolutionConfig( + mode=ResolutionMode.FRONTAL, block_overlap=2, carry_over_length=1 + ) + + +def test_explicit_zero_block_overlap_rejected_outside_sequential() -> None: + # The check is on the keys the user wrote, not on their values: an explicit + # 'block-overlap: 0' is just as ignored as any other value. + with pytest.raises(ValidationError, match="'block-overlap' only applies to mode"): + ResolutionConfig(mode=ResolutionMode.FRONTAL, block_overlap=0) + + +def test_kebab_aliases_are_detected_as_declared() -> None: + with pytest.raises(ValidationError, match="'block-overlap' only applies to mode"): + ResolutionConfig.model_validate( + {"mode": "parallel-subproblems", "block-length": 6, "block-overlap": 0} + ) + + +@pytest.mark.parametrize("mode", _NON_SEQUENTIAL_MODES) +def test_non_sequential_modes_accepted_without_the_fields(mode: ResolutionMode) -> None: + cfg = ResolutionConfig(mode=mode, block_length=10) + assert cfg.block_overlap == 0 + assert cfg.effective_carry_over_length == 0 + + +def test_sequential_mode_still_accepts_both_fields() -> None: + cfg = ResolutionConfig( + mode=ResolutionMode.SEQUENTIAL_SUBPROBLEMS, + block_length=10, + block_overlap=4, + carry_over_length=2, + ) + assert cfg.effective_carry_over_length == 2 diff --git a/tests/unittests/gems_runner/simulation/test_carry_over_initial_values.py b/tests/unittests/gems_runner/simulation/test_carry_over_initial_values.py new file mode 100644 index 00000000..742d29df --- /dev/null +++ b/tests/unittests/gems_runner/simulation/test_carry_over_initial_values.py @@ -0,0 +1,111 @@ +# Copyright (c) 2026, RTE (https://www.rte-france.com) +# +# See AUTHORS.txt +# +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. +# +# SPDX-License-Identifier: MPL-2.0 +# +# This file is part of the Antares project. + +""" +Tests the `initial_values` contract of `build_problem`, i.e. how a carry-over +window from the previous block is turned into constraints of the next one: + +- only time-dependent variables are pinned — a time-independent variable + (`structure.time = False`) is left free in every block; +- a value with no `time` dimension is rejected outright. + +The pinned *window length*, which `carry-over-length` controls through the +session, is covered end to end in +`tests/e2e/functional/test_sequential_carry_over_length.py`. +""" + +import pytest +import xarray as xr + +from gems_craft.expression.expression import literal, param, var +from gems_craft.expression.indexing_structure import IndexingStructure +from gems_craft.model import Constraint, float_parameter, float_variable, model +from gems_craft.study import ConstantData, DataBase, Study, System, create_component +from gems_runner.simulation import TimeBlock, build_problem +from gems_runner.simulation.optimization import _validate_initial_values + +CONSTANT = IndexingStructure(False, False) + + +def _one_time_dependent_one_constant_study() -> Study: + """A single-component study whose model has one time-dependent variable + (`gen`) and one time-independent one (`cap`).""" + plant = model( + id="PLANT", + parameters=[float_parameter("cost", CONSTANT)], + variables=[ + float_variable("gen", lower_bound=literal(0), upper_bound=literal(10)), + float_variable( + "cap", + lower_bound=literal(0), + upper_bound=literal(10), + structure=CONSTANT, + ), + ], + constraints=[ + Constraint(name="Max generation", expression=var("gen") <= var("cap")) + ], + objective_contributions={ + "operational": (param("cost") * var("gen")).time_sum().expec() + }, + ) + database = DataBase() + database.add_data("P", "cost", ConstantData(1)) + system = System("carry_over_contract") + system.add_component(create_component(model=plant, id="P")) + return Study(system, database) + + +def _time_da(values: list[float]) -> xr.DataArray: + """Carry-over array in the shape `_extract_carry_over` produces: a `time` + dimension indexed 0..k-1.""" + return xr.DataArray( + values, dims=["time"], coords={"time": list(range(len(values)))} + ) + + +def test_carry_over_skips_time_independent_variables() -> None: + """Only time-dependent variables are pinned. A time-independent variable + (`structure.time = False`) is left free in every block even when the caller + passes a value for it, so consecutive blocks size it independently.""" + problem = build_problem( + _one_time_dependent_one_constant_study(), + TimeBlock(1, [0, 1, 2]), + [0], + initial_values={ + ("PLANT", "gen"): _time_da([2.0, 3.0]), + ("PLANT", "cap"): _time_da([7.0]), + }, + ) + + constraint_names = set(problem.linopy_model.constraints) + assert "carry_over__PLANT__gen" in constraint_names + assert "carry_over__PLANT__cap" not in constraint_names + + +def test_initial_values_without_time_dim_rejected() -> None: + """A value with no `time` dimension — the shape carried over before + multi-timestep stitching existed — is rejected outright rather than + silently reinterpreted as a single-timestep pin.""" + with pytest.raises(ValueError, match="must carry a 'time' dimension"): + build_problem( + _one_time_dependent_one_constant_study(), + TimeBlock(1, [0, 1, 2]), + [0], + initial_values={("PLANT", "gen"): xr.DataArray(2.0)}, + ) + + # The check is a precondition on the argument, so it does not need a study: + with pytest.raises(ValueError, match="must carry a 'time' dimension"): + _validate_initial_values({("PLANT", "gen"): xr.DataArray(2.0)}) + + assert _validate_initial_values(None) == {} From 0c80b26ebd45a915702910723f6caaf41a45dbec Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 10:11:02 +0000 Subject: [PATCH 4/4] refactor(taxonomy): split library validation out of parsing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses tbittar's review on #278: cross-artifact validation had been put inside parse_yaml_library, which is the reading/validating mixing that #265 set out to undo. - New gems_craft/model/validation.py holds check_library_against_taxonomy (moved unchanged from taxonomy.py, with its _missing helper) plus validate_libraries_against_taxonomy, which absorbs the declared-id and missing-taxonomy checks. - parse_yaml_library goes back to a pure reader: no taxonomy argument, no validation. input_libs likewise loses the argument. - load_study reads input/taxonomy.yml and calls the validation explicitly after parsing, alongside consistency_check. - taxonomy.py no longer imports parsing, so the TYPE_CHECKING guard added to break that cycle is gone — the cycle no longer exists. - Acts on the TODO left by #265: consistency_check moves from resolve_components.py to a new gems_craft/study/validation.py; its callers are repointed and a now-dead Model import is dropped. Conformance is enforced when a study is loaded rather than on every parse, matching how consistency_check and validate_optim_config already work. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DGFvLY7afUMBnwfHgKHHAD --- AGENTS.md | 2 +- docs/CHANGELOG.md | 17 ++- src/gems_craft/model/parsing.py | 24 +--- src/gems_craft/model/taxonomy.py | 91 +----------- src/gems_craft/model/validation.py | 131 ++++++++++++++++++ src/gems_craft/study/folder.py | 6 +- src/gems_craft/study/resolve_components.py | 18 --- src/gems_craft/study/study.py | 4 +- src/gems_craft/study/validation.py | 38 +++++ src/gems_runner/main/main.py | 7 +- tests/e2e/functional/perf_pypsa.py | 2 +- .../test_component_dependent_time_shift.py | 2 +- .../functional/test_libs_yaml_system_yaml.py | 2 +- tests/e2e/functional/test_scenario_builder.py | 2 +- .../lib_parsing/test_taxonomy_check.py | 40 ++++-- .../system_parsing/test_components_parsing.py | 3 +- 16 files changed, 224 insertions(+), 165 deletions(-) create mode 100644 src/gems_craft/model/validation.py create mode 100644 src/gems_craft/study/validation.py diff --git a/AGENTS.md b/AGENTS.md index 600989d0..eef978bc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,7 +72,7 @@ The codebase is split into three packages along a solver-dependency boundary: **`gems_craft/model/`** — Immutable model templates. - `Model`: defines component behavior (parameters, variables, constraints, ports) - `Library`: a collection of models, loaded from YAML -- `Taxonomy` (`taxonomy.py`): categories naming the items a model must expose. Models opt in via `taxonomy-category`; `check_library_against_taxonomy` enforces conformance, called from `parse_yaml_library` for libraries declaring a `taxonomy`. The caller supplies the `Taxonomy`; `load_study` reads it from the optional `input/taxonomy.yml`. +- `Taxonomy` (`taxonomy.py`): categories naming the items a model must expose. Models opt in via `taxonomy-category`. `taxonomy.py` holds the data and `load_taxonomy` only; conformance lives in `validation.py` (`check_library_against_taxonomy`, and `validate_libraries_against_taxonomy` for libraries declaring a `taxonomy`), which `load_study` calls after parsing — `parse_yaml_library` stays a pure reader. `load_study` reads the `Taxonomy` from the optional `input/taxonomy.yml`. - `PortTypeSchema` (`parsing.py`) also declares the hybrid-only `area-connection` (`AreaConnectionSchema`) and `thermal-capacity-connection` (`PortThermalCapacitySchema`) fields; `resolve_library.py`'s `_convert_port_type` parses and discards them for every library, hybrid or not. **`gems_craft/expression/`** — Mathematical expression language and AST (structural/static analysis only — no numeric evaluation). diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 363c7b59..dfe1f081 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -9,9 +9,15 @@ All notable changes to GemsPy are documented here. *N+1*'s first timestep to block *N*'s **last** timestep — a different absolute timestep. - **linopy upgraded to `>=0.9.0`** - the minimum supported Python version rises to **3.11** accordingly (linopy 0.9 requires Python >= 3.11). -- **Breaking** - parsing a library that declares a `taxonomy` raises `ValueError` - if no taxonomy is supplied, or if its id differs from the declared one. +- **Breaking** - loading a study whose library declares a `taxonomy` raises + `ValueError` if no taxonomy is supplied, or if its id differs from the declared + one. - **Breaking** - `TaxonomyData` renamed to `TaxonomySchema`; no alias kept. +- **Breaking** - `check_library_against_taxonomy` moved from + `gems_craft.model.taxonomy` to the new `gems_craft.model.validation`, and + `consistency_check` from `gems_craft.study.resolve_components` to the new + `gems_craft.study.validation`. Reading and validating are now separate modules, + as in `optim_config/`. No behavior change; import paths only. ### Added - **Integer strategy and thermal heuristics** - components can now set @@ -30,9 +36,10 @@ All notable changes to GemsPy are documented here. model-build time; using them inside constraints, binding-constraints, objective contributions, or variable bounds raises a `ValueError`. -- **Taxonomy conformance checked at parse time** - `parse_yaml_library` takes an - optional `taxonomy` and checks every library declaring a `taxonomy` field. - `load_study` reads it from the optional `input/taxonomy.yml`. +- **Taxonomy conformance checked when a study is loaded** - `load_study` reads + the optional `input/taxonomy.yml` and calls + `validate_libraries_against_taxonomy` on every library declaring a `taxonomy` + field. `parse_yaml_library` is unchanged and performs no validation. ### Fixed - **Standard library parsing now accepts hybrid port-type fields** - diff --git a/src/gems_craft/model/parsing.py b/src/gems_craft/model/parsing.py index bbacbed5..cadb383e 100644 --- a/src/gems_craft/model/parsing.py +++ b/src/gems_craft/model/parsing.py @@ -17,7 +17,6 @@ from pydantic import ConfigDict, Field, ValidationError from yaml import safe_dump, safe_load -from gems_craft.model.taxonomy import Taxonomy, check_library_against_taxonomy from gems_craft.utils import ModifiedBaseModel @@ -124,31 +123,12 @@ class LibrarySchema(ModifiedBaseModel): version: Optional[str] = None -def _check_declared_taxonomy( - library: LibrarySchema, taxonomy: Optional[Taxonomy] -) -> None: - """Check a library against the taxonomy it declares conformance to.""" - declared = f"Library '{library.id}' declares taxonomy '{library.taxonomy}'" - if taxonomy is None: - raise ValueError( - f"{declared} but no taxonomy was provided to check it against." - ) - if library.taxonomy != taxonomy.id: - raise ValueError(f"{declared} but was checked against '{taxonomy.id}'.") - check_library_against_taxonomy(library, taxonomy) - - -def parse_yaml_library( - input: typing.TextIO, taxonomy: Optional[Taxonomy] = None -) -> LibrarySchema: +def parse_yaml_library(input: typing.TextIO) -> LibrarySchema: tree = safe_load(input) try: - library = LibrarySchema.model_validate(tree["library"]) + return LibrarySchema.model_validate(tree["library"]) except ValidationError as e: raise ValueError(f"An error occurred during parsing: {e}") - if library.taxonomy is not None: - _check_declared_taxonomy(library, taxonomy) - return library def write_yaml_library(library: LibrarySchema, path: Path) -> None: diff --git a/src/gems_craft/model/taxonomy.py b/src/gems_craft/model/taxonomy.py index 6eba7e18..b2eb2410 100644 --- a/src/gems_craft/model/taxonomy.py +++ b/src/gems_craft/model/taxonomy.py @@ -12,17 +12,13 @@ from dataclasses import dataclass, field from pathlib import Path -from typing import TYPE_CHECKING, Callable, Dict, List, Optional +from typing import List, Optional import yaml from pydantic import Field from gems_craft.utils import ModifiedBaseModel -if TYPE_CHECKING: - # Annotations only — parsing.py imports this module, so this would be circular. - from gems_craft.model.parsing import LibrarySchema, ModelSchema - class TaxonomyItem(ModifiedBaseModel): id: str @@ -63,88 +59,3 @@ def load_taxonomy(taxonomy_file: Path) -> Taxonomy: return Taxonomy( id=data.id, description=data.description, categories=data.categories ) - - -def _missing( - required: List[TaxonomyItem], exposed: List, exposed_key: Callable -) -> List[str]: - """Return the sorted taxonomy item ids not exposed by the model. - - Taxonomy items are always identified by their ``id``; ``exposed_key`` maps each - model-side item to the identifier to compare against (e.g. the ``port.field`` - string for port-field-definitions). - """ - return sorted( - {item.id for item in required} - {exposed_key(item) for item in exposed} - ) - - -def check_library_against_taxonomy( - library: "LibrarySchema", taxonomy: Taxonomy -) -> None: - """ - Validates that every model declaring a taxonomy_category: - 1. References a category that exists in the taxonomy. - 2. Exposes all variables, parameters, ports, port-field-definitions, - constraints, binding-constraints, extra-outputs and properties listed - in that taxonomy category. - - Raises ValueError describing the first violation found. - """ - categories: Dict[str, TaxonomyCategory] = {c.id: c for c in taxonomy.categories} - - by_id: Callable = lambda x: x.id - - # Each entry maps a human-readable field-group name to the required items - # (from the taxonomy category) and the items exposed by the model, plus the - # function identifying a model-side item within that group. Taxonomy items are - # homogeneous (``TaxonomyItem``) and always identified by their ``id``. - def field_groups( - category: TaxonomyCategory, model_schema: "ModelSchema" - ) -> List[tuple]: - port_field_key: Callable = lambda d: f"{d.port}.{d.field}" - return [ - ("variable", category.variables, model_schema.variables, by_id), - ("parameter", category.parameters, model_schema.parameters, by_id), - ("port", category.ports, model_schema.ports, by_id), - ( - "port-field-definition", - category.port_field_definitions, - model_schema.port_field_definitions, - port_field_key, - ), - ("constraint", category.constraints, model_schema.constraints, by_id), - ( - "binding-constraint", - category.binding_constraints, - model_schema.binding_constraints, - by_id, - ), - ( - "extra-output", - category.extra_outputs, - model_schema.extra_outputs or [], - by_id, - ), - ("property", category.properties, model_schema.properties, by_id), - ] - - for model_schema in library.models: - cat_id = model_schema.taxonomy_category - if cat_id is None: - continue - - if cat_id not in categories: - raise ValueError( - f"Model '{model_schema.id}' references taxonomy category '{cat_id}' " - f"which does not exist in taxonomy '{taxonomy.id}'." - ) - - category = categories[cat_id] - for group_name, required, exposed, key in field_groups(category, model_schema): - missing = _missing(required, exposed, key) - if missing: - raise ValueError( - f"Model '{model_schema.id}' (taxonomy-category: '{cat_id}') is " - f"missing {group_name}(s) required by the taxonomy: {missing}." - ) diff --git a/src/gems_craft/model/validation.py b/src/gems_craft/model/validation.py new file mode 100644 index 00000000..6a6ee951 --- /dev/null +++ b/src/gems_craft/model/validation.py @@ -0,0 +1,131 @@ +# Copyright (c) 2026, RTE (https://www.rte-france.com) +# +# See AUTHORS.txt +# +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. +# +# SPDX-License-Identifier: MPL-2.0 +# +# This file is part of the Antares project. + +"""Cross-validation of parsed model libraries. + +Kept apart from `parsing.py` (which only reads YAML into schemas) and from +`taxonomy.py` (which only holds the taxonomy data and reads it from disk), so +that reading and validating stay separate concerns — mirroring +`optim_config/parsing.py` and `optim_config/validation.py`. +""" + +from typing import Callable, Dict, List, Optional + +from gems_craft.model.parsing import LibrarySchema, ModelSchema +from gems_craft.model.taxonomy import Taxonomy, TaxonomyCategory, TaxonomyItem + + +def validate_libraries_against_taxonomy( + libraries: List[LibrarySchema], taxonomy: Optional[Taxonomy] +) -> None: + """Check every library declaring a ``taxonomy`` field against ``taxonomy``. + + Libraries that do not declare a taxonomy are left alone, even when some of + their models carry a ``taxonomy-category``. + + Raises ValueError if a declaring library has no taxonomy to check against, + if it declares a different taxonomy id, or if any of its models violates it. + """ + for library in libraries: + if library.taxonomy is None: + continue + declared = f"Library '{library.id}' declares taxonomy '{library.taxonomy}'" + if taxonomy is None: + raise ValueError( + f"{declared} but no taxonomy was provided to check it against." + ) + if library.taxonomy != taxonomy.id: + raise ValueError(f"{declared} but was checked against '{taxonomy.id}'.") + check_library_against_taxonomy(library, taxonomy) + + +def _missing( + required: List[TaxonomyItem], exposed: List, exposed_key: Callable +) -> List[str]: + """Return the sorted taxonomy item ids not exposed by the model. + + Taxonomy items are always identified by their ``id``; ``exposed_key`` maps each + model-side item to the identifier to compare against (e.g. the ``port.field`` + string for port-field-definitions). + """ + return sorted( + {item.id for item in required} - {exposed_key(item) for item in exposed} + ) + + +def check_library_against_taxonomy(library: LibrarySchema, taxonomy: Taxonomy) -> None: + """ + Validates that every model declaring a taxonomy_category: + 1. References a category that exists in the taxonomy. + 2. Exposes all variables, parameters, ports, port-field-definitions, + constraints, binding-constraints, extra-outputs and properties listed + in that taxonomy category. + + Raises ValueError describing the first violation found. + """ + categories: Dict[str, TaxonomyCategory] = {c.id: c for c in taxonomy.categories} + + by_id: Callable = lambda x: x.id + + # Each entry maps a human-readable field-group name to the required items + # (from the taxonomy category) and the items exposed by the model, plus the + # function identifying a model-side item within that group. Taxonomy items are + # homogeneous (``TaxonomyItem``) and always identified by their ``id``. + def field_groups( + category: TaxonomyCategory, model_schema: ModelSchema + ) -> List[tuple]: + port_field_key: Callable = lambda d: f"{d.port}.{d.field}" + return [ + ("variable", category.variables, model_schema.variables, by_id), + ("parameter", category.parameters, model_schema.parameters, by_id), + ("port", category.ports, model_schema.ports, by_id), + ( + "port-field-definition", + category.port_field_definitions, + model_schema.port_field_definitions, + port_field_key, + ), + ("constraint", category.constraints, model_schema.constraints, by_id), + ( + "binding-constraint", + category.binding_constraints, + model_schema.binding_constraints, + by_id, + ), + ( + "extra-output", + category.extra_outputs, + model_schema.extra_outputs or [], + by_id, + ), + ("property", category.properties, model_schema.properties, by_id), + ] + + for model_schema in library.models: + cat_id = model_schema.taxonomy_category + if cat_id is None: + continue + + if cat_id not in categories: + raise ValueError( + f"Model '{model_schema.id}' references taxonomy category '{cat_id}' " + f"which does not exist in taxonomy '{taxonomy.id}'." + ) + + category = categories[cat_id] + for group_name, required, exposed, key in field_groups(category, model_schema): + missing = _missing(required, exposed, key) + if missing: + raise ValueError( + f"Model '{model_schema.id}' (taxonomy-category: '{cat_id}') is " + f"missing {group_name}(s) required by the taxonomy: {missing}." + ) diff --git a/src/gems_craft/study/folder.py b/src/gems_craft/study/folder.py index 79020138..838b1bba 100644 --- a/src/gems_craft/study/folder.py +++ b/src/gems_craft/study/folder.py @@ -15,14 +15,15 @@ from gems_craft.model.parsing import parse_yaml_library from gems_craft.model.resolve_library import resolve_library from gems_craft.model.taxonomy import load_taxonomy +from gems_craft.model.validation import validate_libraries_against_taxonomy from gems_craft.study.parsing import parse_yaml_system from gems_craft.study.resolve_components import ( build_data_base, - consistency_check, resolve_system, ) from gems_craft.study.scenario_builder import ScenarioBuilder from gems_craft.study.study import Study +from gems_craft.study.validation import consistency_check def load_study(study_dir: Path) -> Study: @@ -50,7 +51,8 @@ def load_study(study_dir: Path) -> Study: input_libraries = [] for lib_file in lib_folder.glob("*.yml"): with lib_file.open() as lib: - input_libraries.append(parse_yaml_library(lib, taxonomy=taxonomy)) + input_libraries.append(parse_yaml_library(lib)) + validate_libraries_against_taxonomy(input_libraries, taxonomy) with system_file.open() as c: input_study = parse_yaml_system(c) diff --git a/src/gems_craft/study/resolve_components.py b/src/gems_craft/study/resolve_components.py index ced3457b..83197607 100644 --- a/src/gems_craft/study/resolve_components.py +++ b/src/gems_craft/study/resolve_components.py @@ -12,7 +12,6 @@ from pathlib import Path from typing import Dict, List, Optional, Tuple, Union -from gems_craft.model import Model from gems_craft.model.library import Library from gems_craft.study import ( Component, @@ -132,23 +131,6 @@ def _get_component_by_id( return components_dict.get(component_id) -# TODO: cross-validation logic mixed into a "resolve" (preprocessing) module — -# move to a dedicated gems_craft/study/validation.py, mirroring optim_config/. -def consistency_check(system: System, input_models: Dict[str, Model]) -> bool: - """ - Checks if all components in the System have a valid model from the library. - Returns True if all components are consistent, raises ValueError otherwise. - """ - # TODO: Update this consistency check to check if each component have a valid model from the lib it refers to (and not all libs) - model_ids_set = input_models.keys() - for component in system.all_components: - if component.model.id not in model_ids_set: - raise ValueError( - f"Error: Component {component.id} has invalid model ID: {component.model.id}" - ) - return True - - def build_data_base( input_system: SystemSchema, timeseries_dir: Optional[Path], diff --git a/src/gems_craft/study/study.py b/src/gems_craft/study/study.py index 32080be0..4c63db00 100644 --- a/src/gems_craft/study/study.py +++ b/src/gems_craft/study/study.py @@ -54,8 +54,8 @@ def models(self) -> Dict[str, Model]: } # TODO: this is a second, disjoint consistency check alongside - # resolve_components.consistency_check() — consider consolidating both - # into one validation module for System/Study. + # study/validation.py's consistency_check() — consider consolidating both + # into that validation module. def check_consistency(self) -> None: """Validate that the database supplies data for every parameter of every component defined in the system. diff --git a/src/gems_craft/study/validation.py b/src/gems_craft/study/validation.py new file mode 100644 index 00000000..924a1486 --- /dev/null +++ b/src/gems_craft/study/validation.py @@ -0,0 +1,38 @@ +# Copyright (c) 2024, RTE (https://www.rte-france.com) +# +# See AUTHORS.txt +# +# This Source Code Form is subject to the terms of the Mozilla Public +# License, v. 2.0. If a copy of the MPL was not distributed with this +# file, You can obtain one at http://mozilla.org/MPL/2.0/. +# +# SPDX-License-Identifier: MPL-2.0 +# +# This file is part of the Antares project. + +"""Cross-validation of a resolved system against the models it refers to. + +Kept apart from `resolve_components.py`, which only resolves the parsed system +into its runtime objects — mirroring `optim_config/parsing.py` and +`optim_config/validation.py`. +""" + +from typing import Dict + +from gems_craft.model import Model +from gems_craft.study.system import System + + +def consistency_check(system: System, input_models: Dict[str, Model]) -> bool: + """ + Checks if all components in the System have a valid model from the library. + Returns True if all components are consistent, raises ValueError otherwise. + """ + # TODO: Update this consistency check to check if each component have a valid model from the lib it refers to (and not all libs) + model_ids_set = input_models.keys() + for component in system.all_components: + if component.model.id not in model_ids_set: + raise ValueError( + f"Error: Component {component.id} has invalid model ID: {component.model.id}" + ) + return True diff --git a/src/gems_runner/main/main.py b/src/gems_runner/main/main.py index 93c6be3e..a0d33721 100644 --- a/src/gems_runner/main/main.py +++ b/src/gems_runner/main/main.py @@ -16,7 +16,6 @@ from gems_craft.model.library import Library from gems_craft.model.parsing import parse_yaml_library from gems_craft.model.resolve_library import resolve_library -from gems_craft.model.taxonomy import Taxonomy from gems_craft.optim_config.parsing import OptimConfig from gems_craft.study import Study from gems_craft.study.data import DataBase @@ -27,14 +26,12 @@ from gems_runner.study.runner import run_study -def input_libs( - yaml_lib_paths: List[Path], taxonomy: Optional[Taxonomy] = None -) -> Dict[str, Library]: +def input_libs(yaml_lib_paths: List[Path]) -> Dict[str, Library]: yaml_libraries = [] yaml_library_ids = set() for path in yaml_lib_paths: with path.open("r") as file: - yaml_lib = parse_yaml_library(file, taxonomy=taxonomy) + yaml_lib = parse_yaml_library(file) if yaml_lib.id in yaml_library_ids: raise ValueError(f"The identifier '{yaml_lib.id}' is defined twice") yaml_libraries.append(yaml_lib) diff --git a/tests/e2e/functional/perf_pypsa.py b/tests/e2e/functional/perf_pypsa.py index 50943679..6fd91fe3 100644 --- a/tests/e2e/functional/perf_pypsa.py +++ b/tests/e2e/functional/perf_pypsa.py @@ -12,10 +12,10 @@ from gems_craft.study.parsing import parse_yaml_system from gems_craft.study.resolve_components import ( build_data_base, - consistency_check, resolve_system, ) from gems_craft.study.system import System +from gems_craft.study.validation import consistency_check from gems_runner.simulation import TimeBlock, build_problem diff --git a/tests/e2e/functional/test_component_dependent_time_shift.py b/tests/e2e/functional/test_component_dependent_time_shift.py index f9bd5aaa..070976c5 100644 --- a/tests/e2e/functional/test_component_dependent_time_shift.py +++ b/tests/e2e/functional/test_component_dependent_time_shift.py @@ -100,9 +100,9 @@ from gems_craft.study.parsing import parse_yaml_system from gems_craft.study.resolve_components import ( build_data_base, - consistency_check, resolve_system, ) +from gems_craft.study.validation import consistency_check from gems_runner.simulation import TimeBlock, build_problem from tests.e2e.functional.libs.standard import ( BALANCE_PORT_TYPE, diff --git a/tests/e2e/functional/test_libs_yaml_system_yaml.py b/tests/e2e/functional/test_libs_yaml_system_yaml.py index 1e886d2a..c8d3bad9 100644 --- a/tests/e2e/functional/test_libs_yaml_system_yaml.py +++ b/tests/e2e/functional/test_libs_yaml_system_yaml.py @@ -47,11 +47,11 @@ from gems_craft.study.parsing import SystemSchema, parse_yaml_system from gems_craft.study.resolve_components import ( build_data_base, - consistency_check, resolve_system, ) from gems_craft.study.study import Study from gems_craft.study.system import System +from gems_craft.study.validation import consistency_check from gems_runner.simulation import TimeBlock, build_problem diff --git a/tests/e2e/functional/test_scenario_builder.py b/tests/e2e/functional/test_scenario_builder.py index c3758171..7a20dc1e 100644 --- a/tests/e2e/functional/test_scenario_builder.py +++ b/tests/e2e/functional/test_scenario_builder.py @@ -21,10 +21,10 @@ from gems_craft.study.parsing import parse_yaml_system from gems_craft.study.resolve_components import ( build_data_base, - consistency_check, resolve_system, ) from gems_craft.study.scenario_builder import ScenarioBuilder +from gems_craft.study.validation import consistency_check from gems_runner.simulation import build_problem from gems_runner.simulation.time_block import TimeBlock diff --git a/tests/unittests/gems_craft/lib_parsing/test_taxonomy_check.py b/tests/unittests/gems_craft/lib_parsing/test_taxonomy_check.py index c792f736..44fa3fff 100644 --- a/tests/unittests/gems_craft/lib_parsing/test_taxonomy_check.py +++ b/tests/unittests/gems_craft/lib_parsing/test_taxonomy_check.py @@ -12,7 +12,6 @@ import io from pathlib import Path -from typing import Optional import pytest @@ -21,9 +20,12 @@ Taxonomy, TaxonomyCategory, TaxonomyItem, - check_library_against_taxonomy, load_taxonomy, ) +from gems_craft.model.validation import ( + check_library_against_taxonomy, + validate_libraries_against_taxonomy, +) def _make_taxonomy(*categories: TaxonomyCategory) -> Taxonomy: @@ -34,8 +36,8 @@ def _make_category(cat_id: str, port_ids: list[str]) -> TaxonomyCategory: return TaxonomyCategory(id=cat_id, ports=[TaxonomyItem(id=p) for p in port_ids]) -def _parse_lib(yaml_content: str, taxonomy: Optional[Taxonomy] = None): - return parse_yaml_library(io.StringIO(yaml_content), taxonomy=taxonomy) +def _parse_lib(yaml_content: str): + return parse_yaml_library(io.StringIO(yaml_content)) # --- valid cases --- @@ -421,7 +423,7 @@ def test_model_exposing_all_required_fields_is_valid() -> None: check_library_against_taxonomy(lib, taxonomy) # must not raise -# --- parse_yaml_library wiring --- +# --- validate_libraries_against_taxonomy --- _CONFORMING_LIB = """ library: @@ -458,34 +460,42 @@ def test_model_exposing_all_required_fields_is_valid() -> None: """ -def test_parse_library_declaring_taxonomy_is_checked() -> None: +def test_library_declaring_taxonomy_is_checked() -> None: taxonomy = _make_taxonomy(_make_category("production", ["injection_port"])) - lib = _parse_lib(_CONFORMING_LIB, taxonomy) + lib = _parse_lib(_CONFORMING_LIB) + validate_libraries_against_taxonomy([lib], taxonomy) # must not raise assert lib.taxonomy == "test_taxonomy" -def test_parse_library_declaring_taxonomy_raises_on_violation() -> None: +def test_library_declaring_taxonomy_raises_on_violation() -> None: taxonomy = _make_taxonomy(_make_category("production", ["injection_port"])) with pytest.raises(ValueError, match="injection_port"): - _parse_lib(_VIOLATING_LIB, taxonomy) + validate_libraries_against_taxonomy([_parse_lib(_VIOLATING_LIB)], taxonomy) -def test_parse_library_declaring_taxonomy_without_argument_raises() -> None: +def test_library_declaring_taxonomy_without_argument_raises() -> None: with pytest.raises(ValueError, match="no taxonomy was provided"): - _parse_lib(_CONFORMING_LIB) + validate_libraries_against_taxonomy([_parse_lib(_CONFORMING_LIB)], None) -def test_parse_library_with_mismatched_taxonomy_id_raises() -> None: +def test_library_with_mismatched_taxonomy_id_raises() -> None: taxonomy = Taxonomy( id="other_taxonomy", categories=[_make_category("production", ["injection_port"])], ) with pytest.raises(ValueError, match="other_taxonomy"): - _parse_lib(_CONFORMING_LIB, taxonomy) + validate_libraries_against_taxonomy([_parse_lib(_CONFORMING_LIB)], taxonomy) -def test_parse_library_without_declared_taxonomy_is_not_checked() -> None: +def test_library_without_declared_taxonomy_is_not_checked() -> None: """A model may carry a taxonomy-category without the library opting in.""" taxonomy = _make_taxonomy(_make_category("production", ["injection_port"])) - lib = _parse_lib(_LIB_WITHOUT_TAXONOMY, taxonomy) + lib = _parse_lib(_LIB_WITHOUT_TAXONOMY) + validate_libraries_against_taxonomy([lib], taxonomy) # must not raise assert lib.taxonomy is None + + +def test_parsing_a_declaring_library_does_not_validate_it() -> None: + """Parsing is pure reading: conformance is the caller's business.""" + lib = _parse_lib(_VIOLATING_LIB) # must not raise + assert lib.taxonomy == "test_taxonomy" diff --git a/tests/unittests/gems_craft/system_parsing/test_components_parsing.py b/tests/unittests/gems_craft/system_parsing/test_components_parsing.py index 7173a3fe..50606143 100644 --- a/tests/unittests/gems_craft/system_parsing/test_components_parsing.py +++ b/tests/unittests/gems_craft/system_parsing/test_components_parsing.py @@ -18,7 +18,8 @@ load_input_system, parse_yaml_system, ) -from gems_craft.study.resolve_components import consistency_check, resolve_system +from gems_craft.study.resolve_components import resolve_system +from gems_craft.study.validation import consistency_check COMPO_FILE = Path(__file__).parent / "systems/system.yml"