Repository navigation
Keep retention progressing when the oldest expired runs are held #500
Description
A retention pass always selects the oldest expired, unarchived closed runs. A run that is skipped stays at the front of that order, and the next pass selects it again. When skipped runs fill a pass's limit, retention makes no progress on other eligible runs in that namespace.
Current behavior
Main 4e0f6458d890f4b34f1959c12dd385b2b1ea01f7:
HistoryRetentionEnforcer::expiredRunIdsorders candidates byclosed_atwith a limit and no exclusion for runs that were recently skipped or failed.- The inline pass (
runInlinePass) uses a limit of 1. One skipped run at the head stops inline pruning for that namespace. pruneRuncan skip for reasons that persist until an operator acts, for exampleexternal_payload_storage_driver_unavailableoractivity_policy_backend_unsupported.- The scheduled
history:prunepass uses a limit of 100, so it stalls only when at least that many such runs sit at the head. It does not print skip reasons and exits 0 when it only skipped.
This was found by reading source. It was not reproduced.
Expected
A held or failing run does not prevent pruning of other eligible runs. Operators can see which runs are held and why.
Acceptance
- With a persistently skipped run at the head, inline and scheduled passes still prune eligible runs behind it.
- Skipped and failed runs are reported with run id and reason through the retention status API and
history:pruneoutput. The command exits non-zero, or clearly reports no progress, when it made none. - Skipped runs are retried later without a tight loop on every pass.
- Forever-retention, archived and hold semantics are unchanged.
- Coordinate Rust parity through Replace PHP Server with a compatible Rust runtime and safe in-place upgrade #325.
Activity
rmcdaniel commentedon Oct 21, 2026
Native status (Rust Server, #325). Native retention landed in #595 and follows published PHP 2.5.14. It has the same head-of-line stall described here:
expired_run_idsorders byclosed_at, idwith a limit and excludes nothing that was skipped earlier.Native is more exposed than PHP. It does not clean up external payload objects or message streams yet, so it skips any run that holds either, with
external_payload_cleanup_unavailableormessage_stream_cleanup_unavailable. That skip persists. On a namespace whose oldest expired run has such a reference, the native inline pass stops pruning that namespace.To keep the two servers compatible, I'll implement the same correction in native once this issue settles on a design. The design matters most where it touches shared state. If skipped runs get a durable retry-after or held marker, that becomes a column or table both servers write, and it belongs in the shared schema. If the scan instead excludes held runs with SQL predicates, native can follow without a schema change.
Please record the chosen approach here before implementation. Native will follow it, and #325's port log tracks the gap until then.
rmcdaniel commentedon Nov 4, 2026
Server open-item pass, 2026-11-04.
Owner: Server for native retention progress; General for legacy PHP disposition. The original head-of-line selection finding is not resolved by raising #633's pass limit or by implementing one cleanup driver. A held oldest run must remain held while later eligible runs can progress.
Next source controls put more held/failing runs than the pass limit ahead of eligible runs and use bounded keyset scanning within the original maintenance deadline. Preserve namespace, archive/forever/backup holds, per-run transaction authority and explicit incomplete/skip reasons. Map the source slice to native #632/#325, with no competing background queue or persistent retry controller. General: sequence this integrity/progress slice with that bounded-maintenance contract.
rmcdaniel commentedon Nov 11, 2026
Triage 2026-11-11: raised to P2. Native retention orders by closed_at,id with no skip exclusion, so held runs stall the head (operator_metrics.rs). Next: keyset past skipped runs and report skip reasons; pairs with #633. Owner: server.
rmcdaniel commentedon Nov 11, 2026
Released in native Server 3.5.0 (https://github.com/durable-workflow/server/releases/tag/3.5.0; index sha256:b5776fea2a4627ebc477b3b350caaf600dc220e7ab75aa29d32b70e404fd7a68 on Docker Hub and GHCR; promoted to 3.5/3/latest after corpus and Compose qualification). Upgrading needs the documented stop/backup/schema-upgrade step.
Prune expired runs from the native maintenance loop
Merge pull request durable-workflow#1124 from durable-workflow/fix/se…