Skip to content

Handle dual and reduced cost - #221

Merged
tbittar merged 28 commits into
developfrom
feature/dual-reduced-cost-clean
Jun 11, 2026
Merged

Handle dual and reduced cost#221
tbittar merged 28 commits into
developfrom
feature/dual-reduced-cost-clean

Conversation

@tbittar

@tbittar tbittar commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR adds support for dual variables (constraint shadow prices) and reduced costs (variable marginal costs) as first-class model outputs in GemsPy.

New expression operators

Two new operators are available in model expression strings:

  • dual(constraint_name) — returns the shadow price of a constraint at each time step
  • reduced_cost(variable_name) — returns the reduced cost of a variable at each time step

Both are parsed and validated at model-build time via the visitor pattern (DualNode, ReducedCostNode). ModelIdentifiers now carries the set of constraint names so that dual() arguments can be checked against known constraints.

Simulation table output

SimulationTableBuilder collects and exposes duals and reduced costs alongside other extra outputs:

  • Constraint duals are read from linopy's model.dual dataset; the builder aggregates __eq, __lb, and __ub suffixes into a single value per constraint.
  • Reduced costs require solver-specific extraction because linopy does not expose a unified API for them (see below).

Solver-specific reduced cost support

Linopy provides no generic reduced-cost accessor, so the builder falls back to each solver's native API:

Solver Method Notes
Xpress >= 9.8 solver_model.getLpSol() Returns (x, slack, duals, djs); DJ list used. Checked before getSolution because xpress.problem also exposes getSolution with an incompatible signature.
Gurobi solver_model.getAttr("RC", solver_model.getVars()) Standard Gurobi attribute query.
HiGHS solution.col_dual Column dual from HiGHS solution object.

If the solver model is None or unsupported, reduced costs are silently skipped (returns {}).

Project environment

  • A new solvers optional dependency group (xpress>=9.8, gurobipy>=10.0) is added to pyproject.toml for running solver-specific tests locally with a free licence.
  • CI installs this group (uv sync --group dev --group solvers) so solver-specific tests run in the pipeline.

Pre-commit consistency

  • Removed the redundant --config pyproject.toml argument from the black hook (black discovers pyproject.toml automatically).
  • Simplified the isort hook args to --profile black (removed explicit --filter-files src tests which conflicted with pre-commit's own file filtering).

Test plan

  • pytest tests/unittests/ — unit tests covering new expression nodes, simulation table builder, and fluent accessor API
  • pytest tests/e2e/functional/test_dual_reduced_cost.py — end-to-end tests for studies 10_5, 10_5_1, 10_5_2 (requires solver licence)
  • pre-commit run --all-files — confirm black and isort hooks pass cleanly

Closes #182

🤖 Generated with Claude Code

@tbittar tbittar changed the title WIP : Handle dual and reduced cost Handle dual and reduced cost Jun 9, 2026
@tbittar
tbittar force-pushed the feature/dual-reduced-cost-clean branch from 8305925 to 142153d Compare June 9, 2026 11:30
@tbittar
tbittar changed the base branch from main to develop June 9, 2026 14:23
@tbittar
tbittar force-pushed the feature/dual-reduced-cost-clean branch from 142153d to 79396a5 Compare June 9, 2026 14:34
@tbittar
tbittar force-pushed the feature/dual-reduced-cost-clean branch from 0f5b43f to 176e3ed Compare June 10, 2026 12:20
Comment thread docs/CHANGELOG.md Outdated
Pre-computed xr.DataArray for each PortFieldId of this model.
Keyed by PortFieldId(port_name, field_name).
block_length:
Number of time steps in the current time block.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we update the docstring?

Comment thread src/gems/study/system.py
Comment on lines +60 to +76
def _check_no_dual_rc_sum_connections(
master_ref: PortRef, slave_ref: PortRef, field_name: str
) -> None:
master_model = master_ref.component.model
slave_model = slave_ref.component.model
master_port_id = PortFieldId(port_name=master_ref.port_id, field_name=field_name)
master_def = master_model.port_fields_definitions.get(master_port_id)
if master_def is None or not contains_dual_or_reduced_cost(master_def.definition):
return
for bc in slave_model.binding_constraints.values():
if uses_sum_connections_on(bc.expression, slave_ref.port_id, field_name):
raise ValueError(
f"Port-field definition '{master_port_id}' contains "
f"dual/reduced_cost (non-linear) and cannot be aggregated "
f"via sum_connections in a binding-constraint of model "
f"'{slave_model.id}'."
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure to understand why it is forbidden. The purpose of authorizing dual/rc in port-field is indeed to receive them in other ports?

Once again : I would say that we can just decide on the degree of the expressions (variable bounds, constraint, binding-constraint or objective). This degree should always be <= 1.

Extra-outputs have free degrees.

@tbittar tbittar Jun 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check is done only within the binding constraints, it is not forbidden within port-definitions.
But I agree with your remark, I should rationalize all checks using the degree computation. Moreover I am not sure this is correctly done for max, min, abs and so on. I will open a specific PR to adress your comment and robustify validation tests and also treating issue #226 which is not handled by this PR contrary to your comment. So if this is ok I propose to merge as is and to continue validation work in another PR

@@ -0,0 +1,208 @@
# Copyright (c) 2024, RTE (https://www.rte-france.com)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this file is necessary? Intuitively I would say that if we didn't need it for abs/floor/ceil/max, we don't need it for dual/rc.

I would say that we can just decide on the degree of the expression (variable bounds, constraint, binding-constraint or objective). This degree should always be <= 1.

@tbittar
tbittar force-pushed the feature/dual-reduced-cost-clean branch from 91d6ad6 to 85626cd Compare June 10, 2026 15:55
@tbittar

tbittar commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

Comments handled in #229 , can we merge this one @aoustry ?

@AntaresSimulatorTeam AntaresSimulatorTeam deleted a comment from aoustry Jun 11, 2026
@aoustry
aoustry self-requested a review June 11, 2026 09:36
@tbittar
tbittar merged commit b9b549b into develop Jun 11, 2026
2 checks passed
@tbittar
tbittar deleted the feature/dual-reduced-cost-clean branch June 11, 2026 09:40
tbittar added a commit that referenced this pull request Jun 11, 2026
* feat/ add operators abs & round (#217)

* Add abs and round unary operators (#216)

Two new unary operators that mirror the floor/ceil pattern:
- abs(x): absolute value
- round(x): banker's rounding (round-half-to-even), matching np.round
  and Python 3's built-in round.

Like floor/ceil, both operators have degree 0 when their argument has
degree 0, so they are usable inside constraints, binding-constraints,
objective contributions, and variable lower/upper bounds whenever the
argument is constant (parameters and literals). Inside extra-outputs
they may wrap any expression — including ones depending on decision
variables — since extra-outputs are evaluated as numeric xr.DataArrays
post-solve.

* feat(readme): modernize the design of the readme file (#223)

* Restyle README with modern layout

* Add GEMS favicon next to 'The GEMS framework' heading

* Remove top logo image from README header

* Replace 'no-code' with 'low-code' in README

* Use GEMS favicon in quick-link nav

* Vendor GEMS favicon under docs/images and reference it locally

* Add uv install instructions to README

* Prepare develop/ for release v0.1.2 (#225)

* Prepare release v0.1.2

Bump version from 0.1.1 to 0.1.2 in pyproject.toml and uv.lock, and
finalize the CHANGELOG with the abs/round operators and the README
modernization that landed since 0.1.1.

https://claude.ai/code/session_01AfUVdznMY9SayUVt5f3K4T

* Update v0.1.2 release date to 2026-06-11

https://claude.ai/code/session_01AfUVdznMY9SayUVt5f3K4T

---------

Co-authored-by: Claude <noreply@anthropic.com>

* Fix imports in doc (#227)

* Update agents.md (#228)

* Handle dual and reduced cost (#221)

* Initial commit

* delete unnecessary issue templae

* exclude compatibility file

* remove compatibility file from issue templates

* delete changelog file

* fix step 9 in issue templates

* add notify workflow

* add token for antares legacy converter

* add issue creation for GemsViewsBuilder repo

* fix/Update links to GEMS repo (#219)

* update links to GEMS repo

* transparent pictures

* fix png image link

* scheme -> schema

* remove duplicate ci (#222)

* Add tests

* WIP

* Use visitor pattern

* Use visitor pattern

* Support xpress, gurobi

* Fix solver specific test and mypy

* Formatting

* Pre-commit consistency

* Reorganize tests

* Update docs

* Update xhangelog

* Formatting

* Complete new visitor implementation following rebase

* Fix usage in ports

* Update tests

* Apply suggestion from @aoustry

* update docstring

---------

Co-authored-by: nikolaredstork <nikola.ilic@redstork-solutions.com>
Co-authored-by: Guillaume_RTEi <165799647+GuillaumeMaistre@users.noreply.github.com>
Co-authored-by: Antoine Oustry, PhD <58943406+aoustry@users.noreply.github.com>

* feat(library <-> taxonomy check) (#214)

* feat(properties):  add properties field to support system parsing

* fix(tests): remove unnecessary blank line in test_systemschema_from_file.py

* refactor: remove unused import PortsConnection from resolve_components.py

* feat(schema): add optional taxonomy_category field to ModelSchema and implement corresponding unit test for YAML parsing

* Apply suggestion from @dusanparipovic

* Update gemspy version to 0.1.0 and clean up whitespace in ModelSchema taxonomy_category field

* feat(changelog): add new features for component properties and taxonomy category

- Introduced optional `properties` for components in `system.yml`, allowing key/value pairs that are normalized into a dictionary.
- Added optional `taxonomy-category` field for models in library YAML files, accessible via `ModelSchema.taxonomy_category`.
- Updated documentation to reflect these changes and provided examples in the user guide.

* fix(docs): clarify properties key naming in documentation and tests

- Updated documentation to replace "key" with "id" in the context of component properties in `system.yml`.
- Adjusted test assertions to reflect the change from "key" to "id" for consistency with the updated documentation.
- Ensured that error messages for duplicate properties also refer to "id" instead of "key".

* feat(study): introduce Study class to encapsulate System and DataBase

- Added a new `Study` dataclass that combines `System` and `DataBase`, centralizing consistency checks.
- Updated `build_problem()` and `build_decomposed_problems()` to accept `Study` directly.
- Refactored related functions and tests to utilize the new `Study` structure, ensuring seamless integration.
- Removed redundant parameters and streamlined the API for better clarity and usability.

* feat(changelog): add changelog

* fix(ruff): fix ruff format on new files

* Apply suggestion from @tbittar

Co-authored-by: tbittar <thomas.bittar@rte-france.com>

* Apply suggestion from @tbittar

Co-authored-by: tbittar <thomas.bittar@rte-france.com>

* Apply suggestion from @tbittar

Co-authored-by: tbittar <thomas.bittar@rte-france.com>

* Apply suggestion from @aoustry

* Apply suggestion from @aoustry

* Update CHANGELOG.md

* Update input documentation on duplicate id handling

Clarified that duplicate ids for properties are rejected.

* fix(taxonomy): import ConstraintSchema and PortFieldDefinitionSchema (#230)

Fixes mypy [name-defined] errors on TaxonomyCategory fields.

Co-authored-by: Claude <noreply@anthropic.com>

* Address pending review comments: taxonomy field-group validation + model-declared properties (#231)

* feat(taxonomy/models): validate all taxonomy field groups and model-declared properties

Address pending review comments on PR #214:

- check_library_against_taxonomy now validates every field group declared
  in a taxonomy category (variables, parameters, ports, port-field-definitions,
  constraints, binding-constraints, extra-outputs, properties) instead of ports only.
- Add an optional `properties` list to ModelSchema and Model (declared keys).
  When resolving a component, every property key declared by its model must be
  present in the component's properties; extra undeclared properties are allowed.
- Add per-field-group taxonomy tests and component model-property tests.
- Remove redundant tests in test_systemschema_from_file.py (covered by
  test_components_parsing.py).


* docs(changelog): fold property/taxonomy notes into 0.1.2

Integrate the model-properties and taxonomy field-group changes into the
upcoming 0.1.2 section instead of a separate Unreleased block, merging with
the existing close entries to avoid redundancy.

* test: move model-property library tests to lib_parsing (#232)

The tests asserting that a library exposes model-declared properties
(parse + resolve) only exercise library parsing/resolution and do not
involve system/component resolution, so they belong in the lib_parsing
test file rather than test_components_parsing.py.

---------

Co-authored-by: Antoine Oustry, PhD <58943406+aoustry@users.noreply.github.com>
Co-authored-by: tbittar <thomas.bittar@rte-france.com>

* Feature/robust checks (#229)

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Juliette-Gerbaux <130555142+Juliette-Gerbaux@users.noreply.github.com>
Co-authored-by: tbittar <thomas.bittar@rte-france.com>
Co-authored-by: nikolaredstork <nikola.ilic@redstork-solutions.com>
Co-authored-by: Guillaume_RTEi <165799647+GuillaumeMaistre@users.noreply.github.com>
Co-authored-by: Dušan <dusan.paripovic@rte-international.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: dual, reduced_cost operators

5 participants