Repository navigation
Keep condition replay drift recoverable instead of failing the workflow #104
Description
Verified problem
Published Python SDK 2.5.0 emits fail_workflow with exception_type: NonDeterministicWaitCondition when a recorded condition key or predicate fingerprint differs from current workflow code. Both cases were reproduced independently in a fresh consumer.
Reproduction:
@workflow.defn(name="wait-example")
class Example:
def run(self, ctx):
yield ctx.wait_condition(lambda: False, key="approved")
return "done"Obtain the initial open_condition_wait command and construct its ConditionWaitOpened history event. Change either condition_key or condition_definition_fingerprint, then call replay(Example, history, []). The resulting wire command is:
{"type":"fail_workflow","exception_type":"NonDeterministicWaitCondition"}The published tests explicitly expect this classification. A worker deployment can therefore request a permanent business failure for a replay-integrity problem.
Required outcome
- Classify condition identity drift as a typed replay/task failure, without emitting
fail_workflow. - Keep the existing run and history available for recovery by a corrected worker.
- Expose the mismatched key/fingerprint and execution identity in diagnostics, with bounded retry behavior.
- Keep explicit application failures terminal.
- Add minimal replay-corpus cases for both mismatch forms and a connected published-artifact scenario: bad deployment is rejected, corrected deployment resumes the original run, and there is no
WorkflowFailedevent. - Verify cold worker replacement and ordinary query/update replay classification where these paths share the condition replayer.
Current main 04790ae42e8da0a83f65026f096572b547b7baa8 still contains the behavior. Verification established SDK command emission, not the terminal state of a connected Server run.
Sources: condition implementation, existing mismatch tests.
Activity
rmcdaniel commentedon Oct 10, 2026
The repair is published in Python SDK 2.5.3, source
e88642df5a7109e2b574522380cd52cd24356e77, through #109.Condition key and predicate fingerprint drift now raise a typed replay fault. Workflow and accepted-update tasks stay recoverable, and queries preserve the original cause. Deploy compatible code, then call
await handle.repair()to resume the blocked task. Ordinary application failures retain their existing behavior.Full main qualification and publication pass. Python3.10–3.12, the official corpus and connected checks are green. Main integration passes33 cases with46 conditional skips, including all four new cases. This default protocol1.19 run does not qualify the separate native/cooperative1.20 gate.
A fresh PyPI wheel, SHA256
1405fef54203873f584aae685333885f25684d70e9f529aecf0b114829aca2e3, passes22 focused/corpus controls and all four workflow/update key/fingerprint recovery cases against published Server2.5.16. An actual registry2.5.2 producer opens the original wait. A cold2.5.3 worker with incompatible code is rejected. After compatible code and explicit repair, a cold worker resumes the same task at attempt2. The original run, complete history prefix and accepted-update identity survive, each completion occurs once, and no terminal workflow/update failure is recorded.Website #200 is deployed and its public manifest was checked. Sample App #190 is merged with all18 PR and all20 main checks passing. Its published activity recovery passes45 cases across all nine PHP/Python/Rust directions with actual Python2.5.3 workers. The published image passes native amd64 and arm64 qualification and promotion. Both public registries' main manifests match the qualified immutable tag, index digest
sha256:c746459b2a7e1f295bb3da32e6b6eb27f631d7f4da4acbb1b9beeafd1730f0d9.Delivery and task cleanup are complete. The separate shared/native protocol qualification remains with the Server owner.