Skip to content

[Async stack 2/6] asyncio composition and durable timers - #358

Draft
andystaples wants to merge 6 commits into
andystaples-async-stack-1-coroutine-dispatchfrom
andystaples-async-stack-2-asyncio-composition
Draft

andystaples wants to merge 6 commits into
andystaples-async-stack-1-coroutine-dispatchfrom
andystaples-async-stack-2-asyncio-composition

Conversation

@andystaples

@andystaples andystaples commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Do not merge this PR, or any other PR in this stack, on its own. The stack was split out of the closed #293 to make review easier. Once every PR in the stack is approved, the whole stack will be squash-merged together.

Stack layer 2/6, built on the coroutine dispatch layer (#357).

Summary

This layer adds general asyncio composition and durable timers to the replay-safe event loop that coroutine orchestrators run on.

Task composition

  • Adds _DurableAsyncTask. A task is retired once its result or exception has been observed, so replaying discarded gather batches and handled failures does not keep decoded payloads or tracebacks alive.
  • Tasks hash by creation sequence. TaskGroup iterates a set of tasks to cancel them, so this keeps cancellation order independent of memory addresses; the reason is documented in code.
  • Cancellation callbacks run in a deterministic order, keyed by when each task was created. Retired tasks never reuse that ordering.
  • TaskGroup, gather, wait_for, and asyncio.timeout compose over durable work:
    • Cancelling a local waiter does not cancel the underlying shared durable operation.
    • The actions emitted when a TaskGroup is cancelled stay the same across replays.
  • A waiter finalized after the dispatch has closed (for example, an asyncio.Task constructed directly rather than through asyncio.create_task) no longer raises during garbage collection. Directly constructed tasks are documented as unsupported.

Durable timers

  • call_later and call_at are backed by durable timers and return _DurableTimerHandle, which compares by identity.
  • Deadlines are rounded up to the next microsecond, so a timer never fires before the loop's deadline. Timezone-aware orchestration clocks are normalized.
  • Deadlines already in the past run immediately, and non-finite delays raise an error.
  • Cancelling a handle cancels its durable timer.
  • asyncio.sleep and timeouts use the existing chunking for long timers.

Unsupported work

  • Every asyncio loop API the durable loop does not implement (executors, cross-thread scheduling, sockets, pipes, subprocesses, readers/writers, signal handlers, TLS, sendfile, task factories, exception handlers, and loop lifecycle) fails through one table with an actionable error, instead of a bare NotImplementedError.
  • asyncio.wait and asyncio.as_completed are not rejected at run time; the contract documents them as unsupported because their ordering is not replay-stable.

Docs (docs/async-orchestrators.md)

  • States that successive dispatches can run on different workers and that all workers processing an instance must use the pinned environment.
  • Notes that delays below the resolution of loop.time() are effectively zero, and that tasks must be created with asyncio.create_task or a task group.

Code docs

  • Extends the list of private asyncio hooks in the asyncio_loop.py module docstring, and clarifies the Task.wait() docstring for asyncio.create_task and task groups.

Scope

Left for later layers

Tests

  • Adds tests/durabletask/test_asyncio_orchestrators.py, covering:
    • composition
    • memory retirement
    • shared-wait cancellation
    • timers
    • unsupported work, including every unimplemented loop API
    • fanout scaling

@andystaples
andystaples added this pull request to stack #365 November 11, 2026 20:39
@andystaples andystaples changed the title [Async stack 2/7] asyncio composition and durable timers [Async stack 2/6] asyncio composition and durable timers Nov 11, 2026
@andystaples
andystaples marked this pull request as draft November 12, 2026 16:30
@andystaples
andystaples force-pushed the andystaples-async-stack-2-asyncio-composition branch from 1b5625c to 1e5bf2f Compare November 12, 2026 16:48
Comment thread durabletask/internal/asyncio_loop.py Resolved
andystaples and others added 3 commits November 12, 2026 13:26
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use immutable dispatch-local task creation hashes so native TaskGroup traversal cannot depend on worker-local object addresses before scheduling shared-Future callbacks. Preserve existing identity equality and cancellation-wakeup ordering.

Cover direct and gather-wrapped shared futures, out-of-order activity completion, fresh dispatch executors, and replay in separate worker processes. Document pinned worker environments and existing-history upgrade risks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Convert loop deadlines with exact rational arithmetic so durable timers never fire before the requested loop time, and add exact-oracle regression tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@andystaples
andystaples force-pushed the andystaples-async-stack-2-asyncio-composition branch from 0cfb7b3 to 44eccfc Compare November 12, 2026 20:32
andystaples and others added 3 commits November 12, 2026 15:33
…sync-stack-2-asyncio-composition

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

# Conflicts:
#	docs/async-orchestrators.md
…sync-stack-2-asyncio-composition

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Report every unimplemented asyncio loop API through one actionable error
table, tolerate waiters finalized after dispatch close, use replay-safe
lookups for cancellation ordering, normalize timezone-aware clocks for loop
timers, and document why task hashing is load-bearing and which private
asyncio hooks the loop relies on. Document timer resolution and that tasks
must be created with asyncio.create_task. Remove changelog entries for
unreleased behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant