Run the scheduler constraint tests under both backends

[Flix6x]

Description

Since #2364 the schedulers build the same model twice — once through Pyomo, once directly in HiGHS — so a test of scheduler behaviour only means something under one backend if the two agree, which is exactly what cannot be assumed.

Only test_solver.py was parametrized over solvers. Everything else ran under the configured default and no other.

That is not hypothetical: on #2355 a fix passed on the Pyomo path and failed on the direct one, and test_commitments.py could not have caught it, because it only ever runs under one solver.

• Modules opt in with RUN_UNDER_EACH_SOLVER = True; an autouse fixture in conftest.py does the switching, so no test signatures change
• Enabled on test_group_constraints.py and test_operation_modes.py (27 tests → 54)
• test_solver_coverage.py makes the remaining hole explicit

What is deliberately not enabled, and why

test_commitments.py and test_storage.py build assets with fixed names, so running each test twice in one fixture scope violates generic_asset's unique-name constraint. I tried it; it fails on both backends with UniqueViolation, which is a fixture problem rather than a backend disagreement. Parametrizing them means making those fixtures unique per parameter first — a larger change than this one, and better done on its own.

Rather than leave that silent, test_solver_coverage.py asserts that every planning test module either opts into the matrix or appears on an EXEMPT list with a reason, and that no EXEMPT entry is stale (module gone, or since opted in). Adding a module then forces a deliberate choice instead of quietly widening the gap.

How to test

pytest flexmeasures/data/models/planning/tests

test_group_constraints.py and test_operation_modes.py now report [appsi_highs] / [highspy] variants.

Proven capable of failing (per #2384): removing RUN_UNDER_EACH_SOLVER from a module makes test_scheduler_modules_run_under_each_solver_or_are_exempt fail with that module named; adding a module to EXEMPT while it still opts in makes test_exempt_list_has_no_stale_entries fail.

Further improvements

• Make the DB fixtures in test_commitments.py / test_storage.py unique per parameter, then drop those two from EXEMPT.

Related items

• Add a Pyomo-free scheduling backend (direct highspy model construction) #2364 — introduced the second backend and the default flip.
• Fix EMS-level flow commitment constraints #2355 — the case this would have caught.
• Require new tests to be proven capable of failing #2384 — the mutation-testing convention this PR follows.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33897274 | 📁 Comparing f223676 against latest (fc0b7bd)

  🔍 Preview build  

1 file changed

± api/v3_0.html

[Flix6x]

At least show that the exempt modules would still succeed when all their tests would use the other solver.

[Flix6x]

Good call — "cannot be parametrized" had quietly become "unverified", which is a weaker claim than the situation warranted.

Added a --lp-solver option that pins a whole run to one backend, so the exempt modules can simply be run again under the other one. An environment variable would not have worked here: TestingConfig does not read FLEXMEASURES_LP_SOLVER.

Result — all three exempt modules pass under appsi_highs, the non-default backend:

pytest flexmeasures/data/models/planning/tests/test_commitments.py \
       flexmeasures/data/models/planning/tests/test_storage.py \
       flexmeasures/data/models/planning/tests/test_process.py --lp-solver=appsi_highs
68 passed

So the exemption costs per-test granularity, not coverage: those modules are green under both backends, just not within one run.

I also checked the flag actually bites rather than silently doing nothing — --lp-solver=definitely_not_a_solver fails with Pyomo's UnknownSolver, so the option really does reach solver selection. Without that check, "68 passed" would have been equally consistent with the flag being ignored.

The command is recorded next to the EXEMPT entries so it can be re-run.

[Flix6x]

Reflowed in b59d1444e — you were right, and it was every block I added: "run once per / backend", "under one backend / if the two agree", "set from the / environment". Text only; 319 passed, 3 xfailed.

I've stopped checking this by eye, since eyeballing is what failed twice. The rule is mechanical — inside a multi-line docstring or comment block, every line but the last must end in punctuation — so I checked it that way, and separated hits in lines this PR adds from pre-existing ones. The only remaining hits in added lines are the two continuation lines of an example shell command, which end in a backslash by nature. The pre-existing hits in conftest.py I left alone; reflowing unrelated prose does not belong in this PR.

If it would be useful, that check is about 30 lines and could go in as a pre-commit hook, so the convention stops depending on a reviewer noticing. Happy to open it separately — it would have caught this, #2364's, and #2218's.