Preserve pending durable timers across signal wakeups

[rmcdaniel]

Observed defect

Published Python SDK 2.4.0 reissues an already scheduled durable timer when a signal wakes the workflow. The second timer receives another durable operation sequence and a later deadline. Once both timers fire, the unchanged workflow can no longer replay its following condition wait. A query reports NonDeterministicReplayError because history contains another timer where the workflow expects the condition.

This reproduces on published Server 2.5.5 before any image replacement. The resulting history also fails replay on published Server 2.5.6, with the same installed SDK and workflow code.

Reproduce

Run a normal Python worker with an echo activity and this workflow:

@workflow.defn(name="pending-timer")
class PendingTimer:
    def __init__(self):
        self.total = 0
        self.released = False

    @workflow.signal("increment")
    def increment(self, amount):
        self.total += amount

    @workflow.signal("release")
    def release(self):
        self.released = True

    @workflow.query("total")
    def current(self):
        return self.total

    def run(self, ctx):
        yield ctx.schedule_activity("echo", ["sample"])
        yield ctx.sleep(120)
        yield ctx.wait_condition(lambda: self.released, key="release")
        return self.total

Start the workflow and wait for ActivityCompleted and TimerScheduled. Send increment(7) while the timer is pending. The query returns seven, but raw Server history now contains two timers for the single sleep call. The observed first timer was operation sequence 2. The second was sequence 3, with the same 120-second delay and a deadline 4.68 seconds later. Wait for the original timers to fire, then query or replay from a fresh worker.

The rejection says workflow sequence 3 recorded TimerScheduled and TimerFired, while the current workflow yielded a condition wait. No application code changed. PHP and Rust controls with the same activity/timer sequence each retain one timer.

Required outcome

• A wakeup preserves the pending timer's original identity and deadline. Repeated signals and cold worker replacement must not schedule another timer.
• After the original timer fires, the unchanged workflow reaches its condition wait, remains queryable and completes after release.
• Check already scheduled remote activities and children for the same resubmission problem. Do not duplicate their durable operations or side effects.
• Keep genuine changed-code/command mismatch diagnostics. Do not hide damaged histories by skipping arbitrary timer rows.
• Add focused regression coverage, qualify the published correction against real Server histories and document how already affected histories can be diagnosed and handled.

Found through published cross-language image-continuity qualification. That qualification remains incomplete until this defect is corrected and the workload is rerun.

[rmcdaniel]

Delivered in PR #97 and
Python SDK 2.4.1.

Recorded scalar timers, remote activities and children now remain pending
through signal wakeups and cold replay. Replay validates the recorded operation
before waiting. Changed timer delays, policies, scopes and cleanup authority
still fail explicitly. Local activity preparation retains its existing path.
The new Avro regression fixture preserves the original timer identity.

Exact PR and merged-source CI pass on Python 3.10, 3.11 and 3.12, with 2,455
unit tests and 22 connected integration tests. Lint, mypy, corpus,
package, docs and public boundary checks pass.

• PR qualification
• Merged qualification
• Publication and supported PyPI installs

Installed durable-workflow==2.4.1 from PyPI, with no SDK source overlay,
against published Server 2.5.5
(sha256:1d12cf8ded6d3cca4864d297214e21fe218d94380defc29f7bc0f46b1487a890)
and 2.5.6
(sha256:a6b91cf6545b611a6c726e51af5552c1b026a41ab0e805dd8ba85e1eb511687d).
The workflow completes an echo activity, waits on its original timer, receives
increments of seven and eleven, then finishes after release. It preserves the
original run and committed history, returns the same 204800-byte external
payload and total eighteen, and records one activity completion, one timer
schedule/fire and one workflow completion. Completed and previously queued
Python runs also finish correctly. This includes SIGKILL and fresh workers.
The broader PHP/Rust image experiment needed worker restarts and is not an
uninterrupted-worker upgrade claim.

Already duplicated histories are not silently rewritten. The live
replay reference
explains diagnosis and reconciling external side effects before replacing an
affected run. Test resources are removed, and the merged GitHub branch is gone.