Skip to content

Commit 73cac11

Browse files
andystaplesCopilot
andauthored
Document async orchestrator authoring contract
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 256ffbd commit 73cac11

3 files changed

Lines changed: 309 additions & 6 deletions

File tree

‎docs/async-orchestrators.md‎

Lines changed: 291 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,291 @@
1+
# Async orchestrators
2+
3+
All three packages accept native `async def` orchestrators alongside existing
4+
synchronous generator orchestrators. Both styles use the same history, action,
5+
retry, payload, and entity engine. Register the function normally with
6+
`TaskHubGrpcWorker`, `DurableTaskSchedulerWorker`, or the Functions
7+
`orchestration_trigger` decorator. Functions supports both native two-argument
8+
and compatibility single-argument orchestrators.
9+
10+
```python
11+
import asyncio
12+
from datetime import timedelta
13+
14+
from durabletask import task
15+
16+
17+
async def workflow(ctx: task.AsyncOrchestrationContext, values: list[int]) -> list[int]:
18+
async def branch(value: int) -> int:
19+
first = await ctx.call_activity("increment", input=value, return_type=int)
20+
await ctx.create_timer(timedelta(seconds=1))
21+
return await ctx.call_activity("increment", input=first, return_type=int)
22+
23+
async with asyncio.TaskGroup() as group:
24+
branches = [group.create_task(branch(value)) for value in values]
25+
return [branch.result() for branch in branches]
26+
```
27+
28+
Each branch resumes as its own durable operation completes, including when
29+
multiple completions arrive in one dispatch. It does not wait for an entire
30+
batch before scheduling its next operation.
31+
32+
> [!WARNING]
33+
> Python scopes are not persisted workflow resources. `finally` blocks and async
34+
> context-manager exits can run when an ordinary dispatch suspends, not only
35+
> when the workflow finishes. Keep durable compensation in normal orchestration
36+
> control flow; see [dispatch teardown](#dispatch-teardown-is-not-workflow-cancellation).
37+
38+
## Cold operations and reusable handles
39+
40+
An `async def` orchestrator automatically receives an
41+
`AsyncOrchestrationContext`. Its durable operations return native coroutines.
42+
Generator orchestrators still receive `OrchestrationContext` and keep their
43+
eager durable task handles. Both contexts use the same replay engine.
44+
45+
| Expression | Meaning |
46+
|---|---|
47+
| `ctx.call_activity(...)` | Create a single-use native coroutine; schedule when it starts. |
48+
| `ctx.as_task_context().call_activity(...)` | Schedule eagerly and return a durable handle. |
49+
| `handle.wait()` | Create a native coroutine borrowing an existing handle. |
50+
| `ctx.wait_for_external_event(...)` | Withdraw an owned event wait on cancellation or timeout. |
51+
| `ctx.create_timer(...)` | Own a timer, cancelling its pending local work on cancellation. |
52+
53+
Use the async context with `asyncio.create_task`, `TaskGroup.create_task`, and
54+
ordered `gather`. Constructing a coroutine does not allocate action IDs,
55+
serialize input, or consume buffered events. Cancelling its local task before
56+
its first step starts no durable operation. Mutable inputs are read when the
57+
coroutine executes, and relative timer delays start from that execution point.
58+
Keep or copy inputs accordingly.
59+
60+
Cold coroutines are single-use, like ordinary Python coroutines. For an
61+
explicitly started operation with multiple independent waiters, retain its
62+
durable handle:
63+
64+
```python
65+
operation = ctx.as_task_context().call_activity("quote", input="order-1", return_type=int)
66+
first = asyncio.create_task(operation.wait())
67+
second = asyncio.create_task(operation.wait())
68+
first.cancel()
69+
value = await second
70+
```
71+
72+
Cancelling either borrower does not cancel the underlying operation. Direct
73+
`await operation` remains supported, but `create_task(operation)` is not valid:
74+
an awaitable durable handle is not a coroutine. Do not retry that expression
75+
assuming the eager operation was rolled back.
76+
77+
Async function detection also recognizes `functools.wraps`, partials, and
78+
callable instances; existing registration naming requirements still apply.
79+
Use `functools.wraps` when a synchronous decorator wraps an async orchestrator;
80+
an unmarked synchronous wrapper returning a coroutine is rejected because it
81+
would have received the generator context.
82+
83+
## Azure Functions compatibility
84+
85+
Single-argument async Functions orchestrators receive
86+
`df.AsyncDurableOrchestrationContext`, with the v1-style method and argument
87+
names. Two-argument native orchestrators receive `task.AsyncOrchestrationContext`.
88+
89+
```python
90+
import azure.durable_functions as df
91+
92+
app = df.DFApp()
93+
94+
95+
@app.orchestration_trigger(context_name="context")
96+
async def greetings(context: df.AsyncDurableOrchestrationContext) -> list[str]:
97+
names = context.get_input()
98+
return await context.task_all([
99+
context.call_activity("say_hello", name) for name in names
100+
])
101+
```
102+
103+
`task_all` starts its inputs, waits for all of them, and returns results in
104+
input order, including duplicate references. If an operation fails, the first
105+
observed failure is raised after all have finished. `task_any` returns the
106+
winning `asyncio.Future`, whether it succeeded, failed, or was cancelled,
107+
without cancelling the losers. Await the winner to obtain its result or failure.
108+
An existing future keeps its identity; a coroutine is wrapped in a new task.
109+
Create tasks yourself when you need to identify the winner or later await the
110+
losers. Repeated references within a group are scheduled only once.
111+
112+
On group cancellation, coroutines started by that group are cancelled and
113+
drained, while supplied futures are borrowed and continue running. Remote work
114+
already issued by either kind of operation is not cancelled. The compatibility
115+
context's `as_task_context()` returns its eager v1-style counterpart when
116+
explicit durable handles are needed.
117+
118+
## Supported authoring contract
119+
120+
Await activities, timers, external events, sub-orchestrations, durable
121+
operations via eager `task.when_all` / `task.when_any`, and entity calls/locks
122+
where supported by the provider. Functions compatibility contexts also provide
123+
async `task_all` / `task_any`. Generator and coroutine parents/children may call
124+
one another through durable sub-orchestrations.
125+
126+
The dispatch-local loop supports FIFO ready callbacks, local futures and tasks,
127+
`asyncio.create_task`, `asyncio.gather`, `asyncio.sleep`, `asyncio.TaskGroup`,
128+
and `asyncio.timeout` / `timeout_at`.
129+
`asyncio.wait_for` uses the same task/future scheduling and durable timers.
130+
Third-party async libraries are not part of this compatibility contract, even
131+
when individual code paths happen to use supported asyncio primitives.
132+
133+
Create branches from ordered inputs. Ready work follows FIFO scheduling;
134+
history completions are delivered in history order, draining ready work after
135+
each completion. Task-group cancellation wakeups are ordered by local task
136+
creation, not set/object-hash iteration. Local tasks are dispatch resources,
137+
not new durable instances.
138+
139+
`loop.time()` uses seconds from a fixed UTC epoch and the current orchestration
140+
clock, never machine monotonic time. `loop.call_at` and `call_later` schedule
141+
durable timers for future finite deadlines. Timer conversion rounds up when
142+
necessary so a durable timer does not precede its floating-point deadline.
143+
Core/Functions long timers reuse the existing chunking; Azure Managed retains
144+
its native long timers. Past deadlines are ready callbacks. `sleep(0)` yields
145+
to ready coroutines without a durable timer. Clock advancement and timers
146+
require history events, not elapsed time in the worker.
147+
148+
> [!WARNING]
149+
> This is not a general-purpose asyncio event loop. External network/storage
150+
> I/O, threads, executors, subprocesses, custom/eager task factories, and
151+
> cross-thread callbacks are outside the orchestrator contract. Perform
152+
> nondeterministic work in ordinary activities. Detected unsupported operations
153+
> raise errors, but the SDK cannot detect every unsafe operation, especially
154+
> synchronous side effects or already-completed foreign awaitables.
155+
156+
`asyncio.wait` and `as_completed` are **not supported deterministic composition
157+
APIs**. `wait` returns unordered sets; `as_completed` converts inputs to a set
158+
and can create tasks in hash-dependent order. Pre-completed/tied tasks and
159+
differences between Python implementations can change ordering even when a
160+
particular example executes successfully. Do not iterate sets to schedule
161+
durable actions. Use ordered `gather`, task groups, or durable `when_any` and
162+
explicitly handle the selected durable task. Other asyncio APIs are not
163+
implicitly supported because they happen to execute.
164+
165+
## Failures, timeouts, and cancellation
166+
167+
Durable failures raise the existing SDK exceptions at the await, preserving
168+
failure details, typed results, and retry policies. `task.when_all` still waits
169+
for every durable child before surfacing failure. In contrast,
170+
`asyncio.gather` with default options reports the first failure without
171+
cancelling siblings. Retain and await siblings if their results matter.
172+
`return_exceptions=True` collects their outcomes in input order.
173+
Unobserved local background-task failures are reported during dispatch cleanup;
174+
ordinary exceptions are logged, not silently discarded or automatically joined.
175+
Discarded local-task results and observed-failure tracebacks are released during
176+
replay rather than retained until dispatch cleanup.
177+
178+
Task groups cancel and drain local sibling tasks on failure and use asyncio's
179+
normal exception grouping rules. Python-version-specific exception grouping
180+
and cancel-count behavior remain those of asyncio.
181+
182+
Cancelling a borrower of an existing durable handle, including `handle.wait()`,
183+
detaches that waiter; it does **not** cancel the activity, sub-orchestration,
184+
entity operation, or external-event subscription. Another borrower can still
185+
await the same handle. Keep the handle if you intend to await it again.
186+
Use explicit SDK cancellation when abandoning a cancellable shared timer/event
187+
wait and deliberately affecting every borrower.
188+
189+
The async context owns its private timer and event registrations. For an event
190+
with a deadline, use the owned receive instead of timing out a borrowed handle:
191+
192+
```python
193+
try:
194+
approval = await ctx.wait_for_external_event(
195+
"approval", timeout=timedelta(seconds=30), data_type=bool)
196+
except TimeoutError:
197+
approval = await ctx.wait_for_external_event("approval", data_type=bool)
198+
```
199+
200+
The timed-out registration is withdrawn before the replacement is registered,
201+
so it cannot consume the next approval. The receive also cancels its own
202+
deadline timer when the event wins. A nonpositive timeout polls an already
203+
buffered event and otherwise raises `TimeoutError` without creating a timer.
204+
This ownership does not extend to an
205+
activity, sub-orchestration, or entity call: cancellation cannot stop already
206+
issued remote work. A task-group sibling failure can therefore leave remote
207+
operations running. Retain and join their handles when completion is required
208+
before compensation or releasing application-level capacity.
209+
210+
An entity lock request can still be granted after its local waiter is cancelled.
211+
Abandoning that waiter can leave locks held until the orchestration ends and
212+
block subsequent critical sections and some entity operations. If a timeout is
213+
needed, retain an acquisition task and shield it from the timeout; later await
214+
it and exit the returned lock to release it. Do not start another critical
215+
section while acquisition is pending.
216+
217+
Cancelling a timer callback removes an uncommitted timer when possible, but
218+
cannot undo a timer already committed to history. Neither local nor durable
219+
cancellation implies external side effects were undone.
220+
221+
An unhandled local root-coroutine cancellation fails the orchestration with
222+
`TaskCancelledError`; it does not request scheduler termination or repeatedly
223+
abandon the same deterministic replay. Use the client to terminate the durable
224+
instance. Worker/dispatch cancellation and process-control exceptions continue
225+
to follow the existing worker recovery policy.
226+
227+
## Dispatch teardown is not workflow cancellation
228+
229+
Every dispatch reconstructs the coroutine and its local tasks from history.
230+
After the response and entity-release obligations are snapshotted, teardown
231+
cancels remaining local tasks, drains cooperative cleanup, closes tracked async
232+
generators, and disposes the loop. This includes dispatches that suspend,
233+
continue as new, encounter a history failure, or reject a version.
234+
235+
Nested `finally` blocks and async context-manager exits may run at **every
236+
dispatch boundary**, before workflow completion. Local cleanup can await ready
237+
work such as `sleep(0)`; it must not wait for new durable or external work.
238+
Scheduling/awaiting durable work during async teardown raises an actionable
239+
error. Teardown must cooperate with cancellation; a non-quiescent cleanup
240+
queue is bounded and reported, not allowed to run indefinitely.
241+
242+
Ordinary teardown failures (including `CancelledError`) are logged without
243+
changing the decided response or skipping other cooperative finalizers.
244+
Unexpected non-cancellation `BaseException`
245+
exits retain the existing terminal-failure/recovery policy; an existing primary
246+
failure is preserved. Teardown status/actions do not replace the response.
247+
Held entity-release obligations are captured before teardown for terminal
248+
payload/failure handling.
249+
250+
Keep workflow cleanup that must be durable in **normal orchestration control
251+
flow**, before return or continue-as-new, not solely in a dispatch finalizer.
252+
No cleanup is guaranteed after process crash or forced termination.
253+
Async-generator *helpers* are closed; async-generator *orchestrators*
254+
(`async def` containing `yield`) are rejected.
255+
256+
## Replay and framework adapters
257+
258+
Ordinary async helpers rerun during replay. Their return values are not
259+
automatically checkpointed. There are no local/virtual activities and no
260+
automatic recording or interception of arbitrary await expressions.
261+
`is_replaying` is not a substitute for recording results. Activities remain
262+
at-least-once work: side effects require appropriate idempotency/deduplication.
263+
264+
Framework adapters can compose supported durable operations using async
265+
helpers. Adapters remain responsible for registering ordinary model/tool
266+
activities, encoding/decoding framework payloads, and supplying replay-stable
267+
IDs, clocks, and message metadata. These responsibilities are not handled by
268+
SDK-wide interception. Libraries that require unsupported asyncio APIs or
269+
nondeterministic operations need an appropriate adapter or activity boundary.
270+
271+
Changing an in-flight generator to equivalent coroutine code is safe **only
272+
if its durable history contract is preserved**: action IDs/order, names,
273+
payloads, retries, timer deadlines/chunks, event consumption, and entity
274+
release order. Introducing concurrency, a different join, or a new timer may
275+
change that contract. Test recorded histories and version/drain affected
276+
instances; basic sequential parity tests are not blanket migration safety.
277+
Switching from eager handles to cold operations can also change action order:
278+
construction order is not execution order.
279+
280+
Supporting several Python versions does not guarantee recorded-history
281+
portability between them. Pin the runtime and dependencies for active
282+
instances and validate histories before upgrades. In particular, Python 3.11
283+
and 3.12+ schedule coroutine operands of `asyncio.wait_for()` differently, which
284+
can change durable action order inside concurrent branches. The async context
285+
does not normalize arbitrary standard-library scheduling.
286+
287+
The Functions `orchestrator_generator_wrapper` helper supports direct durable
288+
awaits in synchronous and async mock-based unit tests; it does not simulate the
289+
asyncio scheduler.
290+
Use the real replay executor/provider or the in-memory backend to test task
291+
groups, gather, and durable timeouts.

‎docs/features.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,13 @@ The following features are currently supported:
44

55
## Orchestrations
66

7-
Orchestrators are implemented using ordinary Python functions that take an `OrchestrationContext` as
8-
their first parameter. The `OrchestrationContext` provides APIs for starting child orchestrations,
9-
scheduling activities, and waiting for external events, among other things. Orchestrations are
10-
fault-tolerant and durable, meaning that they can automatically recover from failures and rebuild
11-
their local execution state. Orchestrator functions must be deterministic, meaning that they must
12-
always produce the same output given the same input.
7+
Orchestrators can be generator functions taking an `OrchestrationContext`, or
8+
`async def` functions taking an `AsyncOrchestrationContext`. Both provide APIs
9+
for starting child orchestrations, scheduling activities, and waiting for
10+
external events. Orchestrations are fault-tolerant and durable: they recover
11+
from failures by rebuilding local state through replay. Orchestrator code must
12+
be deterministic. See the [async authoring contract](async-orchestrators.md)
13+
for supported asyncio operations and replay constraints.
1314

1415
### Orchestration versioning
1516

‎docs/supported-patterns.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,13 @@
22

33
The following orchestration patterns are currently supported.
44

5+
Generator and native coroutine orchestrators share these durable patterns.
6+
See the [async orchestrator contract](async-orchestrators.md) for supported
7+
asyncio composition, durable clocks/timeouts, and dispatch-local cleanup.
8+
Async orchestrators receive cold coroutine operations and owned event/timeouts
9+
automatically. Generator orchestrators retain eager handles; async code can
10+
obtain them through `ctx.as_task_context()` and share them via `handle.wait()`.
11+
512
## Function chaining
613

714
An orchestration can chain a sequence of function calls using the following syntax:
@@ -81,6 +88,10 @@ to nested exception groups containing these signals on Python 3.11 and later.
8188
The existing warning-only handling of direct `asyncio.CancelledError` during
8289
forced teardown is unchanged. Other cancellation or process-control exits
8390
propagate after the worker attempts to abandon the accepted dispatch.
91+
Native coroutine authoring distinguishes a cancelled local root task: an
92+
unhandled local cancellation becomes `TaskCancelledError` and fails the
93+
orchestration instead of repeatedly abandoning replay. It does not terminate
94+
the durable instance; see the [async cancellation contract](async-orchestrators.md#failures-timeouts-and-cancellation).
8495

8596
If a completion or abandonment RPC raises, its server-side outcome may be
8697
unknown. The worker logs the uncertainty and closes the dispatch's originating

0 commit comments

Comments
 (0)