Balance internal commodity nodes (heat/steam networks) via first-class balance groups

[Flix6x]

Description

Stacked on #2218. Completes the missing piece of the CHP/factory work: scheduling a multi-commodity factory end-to-end through StorageScheduler.compute().

• device_scheduler gains a balance_groups argument: each group lists the devices of an internal commodity node (a heat or steam network without a grid connection) whose stock-side flows must sum to zero at every time step. This replaces the earlier workaround of overlapping stock groups with a min=max=0 "reference device" (which never worked automatically from a flex-model — see the investigation notes on dev/simulate-factory).
• The StorageScheduler derives balance groups from the flex-config: a non-electricity commodity without energy prices in the flex-context is treated as an internal node (previously this raised Missing consumption price). Electricity still requires a price.
• Converters are described in the flex-model as one entry per commodity port, tied together by a coupling group — so per-commodity balance groups are disjoint and no overlapping group tricks are needed.
• Added changelog item in documentation/changelog.rst
• Docs: a worked converter + internal-node example in the multi-commodity tutorial (tut_converters), cross-referenced from features/scheduling.rst.

Example flex-model

A converter is one flex-model entry per commodity port, tied by a shared coupling name; a commodity with no price in the flex-context becomes an internal node. A CHP (gas → steam + electricity) feeding an internal steam node:

"flex-model": [
  {"sensor": 6, "commodity": "gas",         "coupling": "chp", "coupling-coefficient": 1.0, "consumption-capacity": "20 kW"},
  {"sensor": 7, "commodity": "steam",       "coupling": "chp", "coupling-coefficient": 0.5, "production-capacity": "10 kW"},
  {"sensor": 8, "commodity": "electricity", "coupling": "chp", "coupling-coefficient": 0.3, "production-capacity": "6 kW"}
]
"flex-context": [
  {"commodity": "electricity", "consumption-price": "50 EUR/MWh", "production-price": "50 EUR/MWh"},
  {"commodity": "gas",         "consumption-price": "20 EUR/MWh", "production-price": "20 EUR/MWh"},
  {"commodity": "steam",       "inflexible-consumption": [{"sensor": 10}]}
]

Each kW of gas produces 0.5 kW steam + 0.3 kW electricity, so 20 kW of gas caps steam at 10 kW and electricity at 6 kW. Each port gives exactly one directional capacity, and that is what marks its direction — the opposite direction defaults to zero, so it need not be written out. steam has no price → internal node: its devices (the CHP/steamer producing, the fixed demand consuming) balance each other every step. The full factory (e-heater + boiler → internal heat, steamer → internal steam, CHP → steam + grid electricity) is in the new docs section and in test_factory_chp_dispatch_through_storage_scheduler.

Converter shorthand — how it would map (follow-up, not in this PR)

A first-class converter shorthand would expand into the per-port + coupling form above, e.g.:

{"converter": "chp",
 "gas":         {"sensor": 6, "power-capacity": "20 kW"},
 "steam":       {"sensor": 7, "coupling-coefficient": 0.5},
 "electricity": {"sensor": 8, "coupling-coefficient": 0.3}}

→ expands to the three coupled entries: the first-listed port (gas) is the import-only coupling reference at coefficient 1.0; the remaining ports (steam, electricity) are export-only at their coefficients. This is noted as a possible future improvement to reduce verbosity; the explicit per-port form is what this PR ships.

Both scheduler backends

#2364 landed a second, Pyomo-free backend that builds the HiGHS model directly, and made it the default (FLEXMEASURES_LP_SOLVER = "highspy"). The node balance is modelled in both:

• Pyomo path — node_balance_rule, unchanged from the original commits here.
• Direct HiGHS path — one equality row per node per time step, sum_d(ems_power[d, j]) == 0.

Groups with no devices are dropped during preparation, so they contribute no row at all, matching the Pyomo path's Constraint.Skip. That detail matters on the direct path, which drops rows whose bounds no finite value can satisfy and skips free rows outright: an empty group had to produce no row rather than a degenerate one.

The balance_group_specs collection lives in scheduling_problem.py (the solver-agnostic module from #2364), as a field on SchedulingProblem, so both backends read it from one place.

An internal_commodity_balance scenario in test_highspy_equivalence.py runs a heat node through both backends.

Caveat worth recording. The first version of that scenario was worthless: both heat devices had a free flow band and no cost incentive, so the optimum was zero flow whether or not the balance was enforced — it passed with the balance rows disabled. Pinning the consumer's draw with derivative equals makes the balance the only reason the producer runs, and the scenario now fails correctly when the rows are removed. Any future scenario here needs the same check: a balance over devices that have no reason to move is satisfied trivially, and asserts nothing.

CI: 12/13 checks pass, tests on Python 3.10/3.11/3.12, DCO signed off.

Further improvements

• balance_groups is the fifth device-grouping parameter on device_scheduler, after stock_groups, ems_constraint_groups, the commitment device_group and Combined Heat and Power (CHP) #2218's coupling_groups. All five are ultimately a set of devices plus a kind of constraint over it, each with its own shape and its own spec-collection block. scheduling_problem.py now owns every one of those collection steps, which makes it the natural place to converge them on one membership concept with a menu of constraint kinds. Engine-side counterpart of the "converge site capacity and group capacity?" question in the flex-config placement draft.
• Balance vs coupling should stay distinct, and be documented as such: coupling (Combined Heat and Power (CHP) #2218) fixes members in fixed proportion to a common level; a balance group constrains their sum to zero. Both read as "devices that move together", so the difference is worth spelling out.

How to test

pytest flexmeasures/data/models/planning/tests/test_commitments.py -k factory_chp_dispatch
pytest flexmeasures/data/models/planning/tests/test_storage.py -k factory_chp_dispatch_through_storage_scheduler

The engine-level factory test is parametrized to run in both modes (reference-device stock groups and balance groups) and must produce identical dispatch. The new StorageScheduler test drives the full factory (CHP + gas boiler + e-heater + steamer meeting a fixed 15 kW steam demand from an inflexible sensor) purely from a flex-model and flex-context.

Related Items

Stacked on #2218; continues the work explored on dev/simulate-factory (#2113-era factory simulation).

🤖 Generated with Claude Code

https://claude.ai/code/session_016MLCUiSdXDqDBmg8GbYp1B

[Flix6x]

Direct HiGHS backend support — done in this branch

(Updated: this note originally described what #2364 would require of this branch. That work has since landed here, so it now records what is in rather than what is needed.)

#2364 is merged, and feat/chp has been updated for it (see the corresponding note on #2218). 1db34e3ca merges the updated feat/chp into this branch and does the same one level up.

balance_group_specs moved to the shared module. It is now a field on the SchedulingProblem dataclass produced by prepare_scheduling_problem(), so both backends read it from one place. node_balance_rule in linear_optimization.py is unchanged.

balance_groups is modelled in the direct backend — one equality per node per time step:

sum_d(ems_power[d, j]) == 0

Groups with no devices are dropped during preparation, so they contribute no row, which matches the Pyomo path's Constraint.Skip for that case. That mattered: the direct backend drops rows whose bounds no finite value can satisfy and skips free rows outright, so an empty group had to produce no row rather than a degenerate one.

Equivalence coverage, and a caveat worth recording. An internal_commodity_balance scenario runs a heat node with a producer and a consumer through both backends.

The first version of that scenario was worthless, and only a mutation test caught it: both heat devices had a free flow band and no cost incentive, so the optimum was zero flow whether or not the balance was enforced — it passed with the balance rows disabled. Pinning the consumer's draw with derivative equals makes the balance the only reason the producer runs, and the scenario now fails correctly when the rows are removed.

Worth remembering for any future scenario here: a balance constraint over devices that have no reason to move is satisfied trivially, so the test asserts nothing. Check that a new scenario fails when you disable the thing it is meant to cover.

CI is green: 12/13 checks pass, tests on Python 3.10/3.11/3.12, with DCO now signed off.

[Flix6x]

Please also check the tests against the new mutation-test policy.

[Flix6x]

Now that we support group-level commitments, it seems like a small step to support internal-node commitments, too. This might be used to model wear and tear of an internal system, for instance, steam pipe maintainance.

Not sure if there are internal nodes where such costs are actually significant to take into account when scheduling, though.

[Flix6x]

I went to build this, and it does not work in the obvious form — worth writing down before anyone tries again.

A commitment binds quantity + deviations − sum(ems_power over its devices). An internal node's devices are pinned by node_balance_rule to sum_d(ems_power[d, j]) == 0. So a commitment scoped to a node's own device set has a summation term that is identically zero: its deviations are forced to −quantity regardless of the schedule, giving a constant cost that cannot influence any decision.

Checked rather than reasoned about. A heat node with a free producer and a consumer pinned at −0.4, with a flow commitment over both node devices:

wear price 99   term=optimal   costs=0.000   producer=[0.4, 0.4, 0.4, 0.4]
wear price 0    term=optimal   costs=0.000   producer=[0.4, 0.4, 0.4, 0.4]

Identical schedule and identical cost at wear prices of 99 and 0. The commitment binds nothing — exactly the vacuous-test shape #2384 is about, except here it would have been a vacuous feature.

What wear and tear actually needs is throughput, not net flow. The cost of running steam through a pipe scales with how much goes through it, and the node's net is zero by construction precisely because everything produced is consumed. The quantity to price is the one-sided sum — total production into the node, or equivalently total consumption from it.

Two ways to get there:

1. Available today, no new feature. Scope a commitment to the producing devices of the node via the existing sensors scope. Their summed flow is the node's throughput, and it is a genuine decision variable. This works now and needs nothing from us.
2. A one-sided or throughput commitment kind, which would be a real addition to the commitments framework rather than a scope change — the deviation variables currently model a signed two-sided band, and throughput is |flow|.

Given (1) covers the steam-pipe case, I would not build (2) on the strength of a hypothetical. You flagged your own uncertainty about whether such costs matter to scheduling; I would leave it until a scenario needs it, and this note stands as the reason the naive version was not shipped.

Happy to add a short docs note under the internal-nodes section saying "to price throughput, scope a commitment to the producing side" if that is useful.

[Flix6x]

Yes, please add that note.

[Flix6x]

Added, in the internal-nodes section of the tutorial:

Pricing what flows through a node. An internal node's flows sum to zero by construction, so a commitment scoped to the node itself would bind nothing: there is no net flow to price.
To put a cost on throughput — pipe wear, say, or a conversion levy — scope the commitment to the devices producing into the node, whose summed flow is what actually passes through it.

Placed with the internal-node explanation rather than under commitments, since that is where a reader meets the zero-sum property and is most likely to reach for the thing that does not work.

[Flix6x]

Mutation-test audit (per #2384)

Every test this PR adds, checked by breaking what it covers and confirming it goes red. Three mutations, each reverted after.

1. Node balance disabled (node_balance_rule skipped in Pyomo, balance rows skipped in the direct backend):

FAILED test_factory_chp_dispatch[True]
FAILED test_factory_chp_dispatch_through_storage_scheduler

test_factory_chp_dispatch[False] correctly survives — that is the stock-groups mode, which does not use balance groups. test_commodity_flex_context_internal_node_flag also survives, being a schema test.

2. Internal-node detection disabled (is_internal_node forced to False):

FAILED test_commodity_flex_context_internal_node_flag[context_input0-True]
FAILED test_factory_chp_dispatch_through_storage_scheduler

The engine-level test survives, as it passes balance groups directly rather than deriving them from a flex-context.

3. Coupling disabled (coupling_device_specs skipped in both backends):

FAILED test_factory_chp_dispatch[False]
FAILED test_factory_chp_dispatch[True]
FAILED test_factory_chp_dispatch_through_storage_scheduler

This one was specifically to check that [False] and the dispatch assertions bind at all — mutation 1 leaves [False] passing by design, so on its own it says nothing about whether that parametrization asserts anything. It does.

Result: no vacuous tests. Each of the four fails under at least one mutation, and each survival above is explained by what the test is scoped to rather than by the test being empty.

Already checked earlier in the same way: scenario_internal_commodity_balance (which was vacuous when first written, and was fixed), test_db_flex_model_accepts_an_internal_commodity and test_tutorial_chp_example_validates.