Catch up on missed automation runs

[Flix6x]

Problem

flexmeasures jobs run-automations was stateless: if the host's cron missed a minute (downtime, overload), automations due in that minute were silently skipped forever.

Solution

• The runner now remembers the last processed minute in Redis (key automation-runner:last-tick, set after each invocation completes). On each invocation, it processes every minute from just after the last tick through the current minute.
• A new --max-catchup option (in minutes, default 60) bounds how far back the runner looks; 0 disables catching up. Without a recorded last tick (first run), only the current minute is processed.
• An automation due in several missed minutes still runs only once per invocation: its timing parameters (e.g. forecast start) are resolved against the current time, so queueing multiple identical jobs would be wasteful.
• The per-(automation, minute) Redis guard keys are kept, and now acquired for every due minute of an automation, so concurrent invocations (also with overlapping catch-up windows) cannot double-queue jobs.

Changes

• flexmeasures/data/services/automations.py: new get_due_automations_in_window(start, end) returning each due automation once, with its due minutes (DST-safe minute iteration); get_due_automations now delegates to it.
• flexmeasures/cli/jobs.py: catch-up window computation, --max-catchup option, last-tick bookkeeping, updated help text.
• Docs: catch-up semantics in the "Automating forecasts" section, commands.rst one-liner, CLI changelog bullet.
• Tests: catch-up queues a missed automation once, once-per-invocation for an every-minute cron, --max-catchup bounding (incl. 0 disabling catch-up), and first-run behavior.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #33544972 | 📁 Comparing 86fccf3 against latest (60cfda8)

  🔍 Preview build  

17 files changed · + 3 added · ± 14 modified

+ Added

• _autosummary/flexmeasures.data.models.automations.html
• _autosummary/flexmeasures.data.schemas.automations.html
• _autosummary/flexmeasures.data.services.automations.html

± Modified

• changelog.html
• genindex.html
• http-routingtable.html
• py-modindex.html
• _autosummary/flexmeasures.api.v3_0.assets.html
• _autosummary/flexmeasures.data.models.data_sources.html
• _autosummary/flexmeasures.data.models.html
• _autosummary/flexmeasures.data.schemas.html
• _autosummary/flexmeasures.data.services.html
• _autosummary/flexmeasures.ui.views.assets.views.html
• and 4 more...

[BelhsanHmida]

Closing this in favour of #2396, which has now been merged into #2290.

I checked what would be lost, and the catch-up behaviour is covered:

• catching up after downtime — test_run_automations_catches_up_once_after_downtime
• several missed occurrences running only once — test_missed_occurrences_are_coalesced
• starting cleanly with no prior state — test_persisted_cursor_survives_restart
• the behaviour is documented in documentation/features/forecasting.rst and the CLI change log

The two approaches differ in where the state lives, and that is the main reason this one is the
harder to keep. This PR remembers one global automation-runner:last-tick in Redis for the whole
runner; #2396 stores a scheduling cursor per automation in the database. The per-automation cursor
means a newly created automation starts from its own creation rather than inheriting the runner's
history, it survives losing Redis, and it lets an edit to an automation's recurrence rebase only
that automation. Because the two models disagree about what state exists, get_due_automations
ends up with different signatures in each, so they cannot both land.

One idea from here does survive, and I did not want it lost with the PR: --max-catchup, and in
particular --max-catchup 0 to skip catch-up entirely. #2396 has no operator control over catch-up
at all. The flood protection that originally motivated the option is no longer needed, since missed
occurrences are coalesced into one job and its timing parameters resolve against run time, so the
caught-up run produces a current forecast rather than a stale one. But an operator coming back from
a long outage may still want to skip the missed run and simply resume, and today the only way is to
deactivate and reactivate every automation. I have filed that separately as #2423.