fix(engine): preserve failed reaper batches with delayed retries

[mbuckton]

A failed interest lookup or removal can abandon a drained completion batch, and an exception escaping the periodic callback can cancel future reaper ticks. This change adds recovery around the serialized/coalesced reaper already in development (MSG-413).

• Hold the first failed ID and continue removing healthy IDs.
• Stop on the second failure in the same pass, including across time slices; retain failed and unprocessed IDs for delayed retry.
• Wait at least five seconds before retrying, recheck subscription interest and retained state, and exclude successful removals from retry.
• Make already queued work respect a later failure pause; prevent duplicate completions from bypassing a held ID's delay.
• Preserve batches when lookup or submission fails, reset scheduling state, and keep periodic scheduling alive after rejection.

This is the targeted follow-up to MSG-414, not a merge of the older fork implementation in #2286. The existing serialized interest snapshot and coalesced scheduling are retained.

Validation: merged development commit c16a1786c7463612a50b2c07f9a9a381a683b2c8 passed Jenkins #245: 8,824 passed, 0 failed, 38 skipped (8,862 total). ReaperRetryQueueTest: 10 passed; BulkRemoveMessageTaskFailureTest: 7 passed; DestinationReaperInterleavingTest: 15 passed; EventReaperQueueTest: 2 passed; DestinationImplMessageOperationCoverageTest: 24 passed. The existing MSG-413 interleaving cases are retained. Local standalone retry tests and git diff --check also passed; local Maven was dependency-blocked, superseded by the successful development CI run.

Refs: MSG-414

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: f65b50fa-cfc5-4933-a9ce-0d3b818f4599

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}