Add a Pyomo-free scheduling backend (direct highspy model construction)

[Flix6x]

Description

Building the scheduling model through Pyomo dominates scheduling-job time: profiled on a large co-simulation database, the Pyomo layer (expression-tree construction + appsi ingestion) accounted for ~1 s of a battery-only job and ~7 s of a two-device job with operation-mode power bands, while the solve itself finishes in seconds and belief reads are negligible (~0.3 s). A prototype building the identical battery model via highspy's array API lands at ~5 ms.

This PR adds a Pyomo-free backend that constructs the HiGHS model directly, and factors out the input handling both backends share.

What's in it

• New module flexmeasures/data/models/planning/highspy_optimization.py — a faithful reimplementation of device_scheduler's model semantics (device power up/down with efficiencies, stock dynamics incl. storage efficiency / stock delta / gain / usage, stock bounds and targets, device and EMS power capacities, the commitments framework incl. grouped equalities, EMS-level flow commitments and subcommitments) using vectorized numpy + highspy. The module docstring notes the two-models-in-sync maintenance trade-off and points at device_scheduler as the semantic reference.

• New module flexmeasures/data/models/planning/scheduling_problem.py — the solver-agnostic half, which both backends now share. The two backends necessarily build the same model in two representations, but everything around that had been written twice: 199 lines were byte-identical between the two files. Argument normalisation, stock groups and their validation, legacy commitment conversion, the sub-commitment split, device_group_lookup, the convex-cost-curve check, the Big-Ms, band validation, the HiGHS option profile, and the cost/schedule assembly have no solver in them. prepare_scheduling_problem() returns a SchedulingProblem that both backends unpack. Duplication between the backends is now 40 lines, and those 40 are the shared signature and the call itself.

• Exposed as a solver choice: FLEXMEASURES_LP_SOLVER = "highspy". device_scheduler dispatches to the new module for that value; every other value keeps the Pyomo path untouched. Same inputs, same 4-tuple return contract (results/model shims cover every consumer usage: termination_condition, commitment_costs, indexed power views).

• Arguments are forwarded by name, not by a hand-written keyword list. With a hand-written list, whoever adds the next device_scheduler parameter (e.g. coupling_groups in Combined Heat and Power (CHP) #2218, balance_groups in Balance internal commodity nodes (heat/steam networks) via first-class balance groups #2289) naturally works on the Pyomo model, and a parameter missing from that list would not fail — it would simply never reach the backend, producing a schedule computed as if the constraint had never been requested. Since this PR makes highspy the default, that would be silently wrong. Forwarding by name removes the failure mode; an argument the backend does not model raises NotImplementedError naming it, and a test asserts the backend covers the whole current signature.

• CI equivalence: the app_with_each_solver fixture now runs ["appsi_highs", "cbc", "highspy"], and test_highspy_equivalence.py asserts near-identical schedules across both HiGHS paths on seven representative scenarios (battery+prices, soc-targets with storage efficiency and stock delta, site capacity with breach/peak prices, two devices with a StockCommitment, an EMS-level flow commitment, an EMS-level commitment scoped to one commodity, and an infeasible case).

• Default flipped to highspy (config_defaults), with docs updated (configuration, deployment, installation).

Local verification: 318 passed, 3 xfailed across the planning suite under all three solver params (cbc installed, nothing skipped), plus the scheduling-job and API scheduling tests; mypy/black/flake8 clean.

Performance

Two measurements on a 4-device × 192-step problem, beyond the Pyomo-bypass win above:

• Sub-commitments are usually one row each (every time step tends to form its own commitment group), so device_group_lookup was slicing a fresh two-column DataFrame, running dropna() on it and calling iterrows(), once per time step. Reading the two columns as arrays instead takes that loop from 49 ms to 1.0 ms (device+device_group) and 65 ms to 0.8 ms (device only).
• Together with the earlier sub-commitment work, prepare_scheduling_problem as a whole went from ~135 ms to ~25 ms, against a model build measured in single-digit milliseconds. Input prep, not model construction, was the remaining bottleneck on the fast path — and it is now in one shared place, so both backends benefit.

The by-name argument forwarding costs ~2 µs per device_scheduler call (the signature comparison is cached on the two function objects; ~70 µs once per process).

Discussion points (deliberate deviations, called out for review)

1. Degenerate optima may tie-break differently between backends. test_multiple_devices_simultaneous_scheduler case 2 asserted one specific optimum of a problem whose per-device allocation is non-unique (same total cost either way); it now asserts solver-independent properties (aggregate schedule, total cost, total unmet demand). Case 1 (unique optimum) keeps exact assertions.
2. Rows whose bounds no finite value can satisfy (upper bound of -inf, or lower bound of +inf, as happens when a commitment quantity is ±inf) are skipped. On the Pyomo path such rows are rejected by HiGHS' addRow when appsi adds them, so the net effect is the same.
3. Solver results and model objects are lightweight shims rather than Pyomo objects, exposing the attributes callers actually consume.
4. An empty commitments list is now tolerated. The Pyomo path used to crash on pd.concat([]) where the direct path coped; sharing the input handling fixes it once, for both.
5. initial_stock_of() casts its device index to int, as the direct backend already did. The Pyomo version raised TypeError on the numpy float device indices a commitment's "device" column can carry.
6. commodity_devices is computed lazily (a cached_property). It is a per-row scan that only the EMS-level flow commitment constraints need, so computing it eagerly would put a real cost on the fast path.
7. There is no solver time-limit knob on main; operators can pass time_limit via FLEXMEASURES_LP_SOLVER_OPTIONS (applied last on both paths).

Note on #2355

An earlier revision of this PR did not build ems_flow_commitment_equalities, because on the Pyomo path those rows had no bounds and HiGHS dropped them anyway. #2355 gave them the same one-sided bounds grouped_commitment_equalities uses, so they bind now, and this PR builds them — otherwise an EMS-level commitment would be silently ignored under the new default backend. The row is the grouped one with a different summation set (all devices, or the commitment's commodity's devices), so both constraint families now share one implementation.

How to test

pytest flexmeasures/data/models/planning/tests (runs all three solver params), or set FLEXMEASURES_LP_SOLVER="appsi_highs" to keep the previous default behavior.

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33890375 | 📁 Comparing bc0f13d against latest (45864fd)

  🔍 Preview build  

54 files changed · + 7 added · ± 47 modified

+ Added

• _autodoc/INFLEXIBLE_CONSUMPTION.html
• _autodoc/INFLEXIBLE_PRODUCTION.html
• _autosummary/flexmeasures.api.common.rate_limiting.html
• _autosummary/flexmeasures.data.models.planning.highspy_optimization.html
• _autosummary/flexmeasures.data.models.planning.scheduling_problem.html
• _autosummary/flexmeasures.data.schemas.scheduling.groups.html
• host/plans.html

± Modified

• changelog.html
• configuration.html
• genindex.html
• index.html
• py-modindex.html
• _autodoc/GROUP.html
• _autodoc/INFLEXIBLE_DEVICE_SENSORS.html
• _autosummary/flexmeasures.api.common.html
• _autosummary/flexmeasures.api.common.responses.html
• _autosummary/flexmeasures.api.common.schemas.generic_schemas.html
• and 37 more...

[Copilot]

Pull request overview

Adds a new scheduling backend that bypasses Pyomo and constructs the equivalent optimization model directly with highspy (HiGHS), then makes it selectable via FLEXMEASURES_LP_SOLVER="highspy" and flips that to the default. This aims to substantially reduce scheduling-job overhead dominated by Pyomo model construction / ingestion.

Changes:

• Introduce flexmeasures/data/models/planning/highspy_optimization.py implementing the device scheduler model directly in HiGHS (with lightweight result/model shims).
• Add solver dispatch in device_scheduler, expand the solver-parametrized planning test fixture, and add equivalence tests comparing appsi_highs vs highspy.
• Update docs + changelog to reflect the new default solver backend and installation/deployment guidance.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
flexmeasures/utils/config_defaults.py	Switch default LP solver config to highspy
flexmeasures/data/models/planning/linear_optimization.py	Dispatch to the new highspy backend when configured
flexmeasures/data/models/planning/highspy_optimization.py	New direct-HiGHS scheduler model implementation + result/model shims
flexmeasures/data/models/planning/tests/conftest.py	Run planning tests under appsi_highs, cbc, and highspy
flexmeasures/data/models/planning/tests/test_solver.py	Make a previously solver-tie-broken test assert solver-independent properties
flexmeasures/data/models/planning/tests/test_highspy_equivalence.py	New scenario-based equivalence tests for appsi_highs vs highspy
documentation/configuration.rst	Document the backend split (highspy direct vs Pyomo solver interfaces) and new default
documentation/host/installation.rst	Update installation guidance to reflect highspy as the default backend
documentation/host/deployment.rst	Update deployment guidance for solver installation under the new default
documentation/changelog.rst	Add changelog entry announcing the new backend and default change

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (9)

flexmeasures/data/models/planning/scheduling_problem.py:40

• This docstring breaks lines mid-phrase (e.g. after "HiGHS'"). Please reflow so each line ends after punctuation (or keep sentences on one line) to follow the repo's docstring wrapping convention.

    """Raise if HiGHS would refuse any of these options.

    Pyomo's appsi_highs interface applies solver options without checking HiGHS'
    return status, so an unknown name, an invalid value, or a feature missing from
    the installed HiGHS build is otherwise ignored without a word. That silently

flexmeasures/data/models/planning/scheduling_problem.py:111

• The convert_commitments_to_subcommitments docstring wraps lines mid-phrase (e.g. line breaks inside a sentence without ending punctuation). Please reflow to keep sentence breaks aligned with line breaks.

    """Transform commitments, each specifying a group for each time step, to sub-commitments, one per group.

    'Groups' are a commitment concept (grouping time slots of a commitment),
    making it possible that deviations/breaches can be accounted for properly within this group
    (e.g. highest breach per calendar month defines the penalty).

flexmeasures/data/models/planning/linear_optimization.py:88

• This docstring wraps mid-phrase in several places (e.g. breaking lines after "next" and before the rest of the sentence). Please reflow so each physical line ends after punctuation, or keep full sentences on one line.

    """Map ``device_scheduler``'s arguments onto the direct HiGHS backend's signature.

    A hand-written keyword list here would be a trap: whoever adds the next
    ``device_scheduler`` parameter naturally works on the Pyomo model further
    down this file, and a parameter missing from that list would not fail — it

flexmeasures/data/models/planning/highspy_optimization.py:22

• The remainder of the module docstring wraps mid-phrase (e.g. the "Deviations" paragraph and list items). Please reflow so line breaks occur after punctuation, while keeping list indentation intact.

Deviations from the Pyomo implementation (all verified against the behavior of
the ``appsi_highs`` path):

- Rows whose computed bounds are impossible to satisfy for any finite value
  (upper bound of -inf, or lower bound of +inf, as happens when a commitment

flexmeasures/data/models/planning/tests/test_highspy_equivalence.py:11

• Module docstring wraps lines mid-phrase. In this repo, docstrings/comments should not be broken in the middle of a phrase; reflow so each physical line ends after punctuation (or keep sentences on one line) to keep review comments and text search stable.

"""Equivalence tests for the direct highspy scheduling backend.

Each scenario is run through ``device_scheduler`` twice: once with the Pyomo
path (``appsi_highs``) and once with the direct HiGHS path (``highspy``), and
the resulting schedules, costs and termination handling are compared.

flexmeasures/data/models/planning/scheduling_problem.py:16

• Module docstring wraps several lines mid-phrase (e.g. breaking right after the function reference and after conjunctions). Please reflow it so each physical line ends after punctuation (or use longer lines) to match the repo's docstring/comment wrapping convention.

This issue also appears in the following locations of the same file:

• line 36
• line 107

"""Solver-agnostic preparation of the device scheduler's inputs.

:func:`flexmeasures.data.models.planning.linear_optimization.device_scheduler`
(Pyomo) and
:func:`flexmeasures.data.models.planning.highspy_optimization.device_scheduler_highspy`

flexmeasures/data/models/planning/tests/test_solver.py:3163

• New inline comments wrap mid-phrase across multiple lines (e.g. splitting a clause without ending punctuation). Please reflow these comments so each physical line ends after punctuation to match the repo's comment-wrapping convention.

    # Expected results with unfair unmet demand and not entirely unfair costs.
    # NB This problem has multiple optima: only the site-level (aggregate)
    # schedule is unique, while the per-device allocation of the charging slots
    # (and thereby the per-device costs and even which device's demand goes
    # unmet) is an arbitrary tie-break that depends on the solver backend.

flexmeasures/data/models/planning/linear_optimization.py:51

• This docstring wraps mid-phrase (line breaks inside a sentence without punctuation at the line end). Please reflow to keep line breaks aligned with punctuation, as per the repo's docstring/comment wrapping convention.

This issue also appears on line 84 of the same file.

    """Whether an argument was left at its default.

    Best-effort: pandas values compare element-wise, so ``value == default`` may
    return an array (or raise) rather than a bool. Anything we cannot decide is
    reported as "not the default", which errs towards raising in

flexmeasures/data/models/planning/highspy_optimization.py:17

• The module docstring's warning section wraps lines mid-phrase (several lines end without punctuation). Please reflow to follow the repo's docstring/comment line-break convention, while keeping the RST directive indentation intact.

This issue also appears on line 18 of the same file.

"""Direct HiGHS (highspy) implementation of the device scheduler.

.. warning:: TWO MODELS TO KEEP IN SYNC

    This module deliberately duplicates the mathematical model of