diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..90f2806b --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,7 @@ +# Summary + +## Serialization compatibility + +- [ ] If this changes default serialization, I reviewed the serialization + compatibility policy in `CONTRIBUTING.md` and described the replay impact + and any applicable exception. Otherwise, this item does not apply. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index b7174508..f86ab820 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -80,6 +80,22 @@ Examples: - Add a regression test that confirms a new concrete API is absent from the public ABC's `__abstractmethods__`. +## Serialization Compatibility + +- Follow the [serialization policy](../docs/serialization.md) for changes + to default serialization, including provider adapters. Use the affected + provider's documented contract to establish existing support. +- Preserve encoded payloads, reconstructed values, raw fallback, and failure + behavior between major releases, subject to the exceptions below. Defer + consideration of new types and unsupported edge cases until a breaking major + release rather than adding feature-specific opt-in flags. +- Non-major replay-breaking changes are limited to security fixes and repairs + to previously supported behavior. An unsupported annotation is not supported + merely because a related type is supported or its raw value is returned. +- For an allowed replay-impacting change, add focused regression coverage, + including replay where appropriate, and document the risk and upgrade + mitigation. A previous failure can itself affect replay if it is caught. + ## Copyright Headers Every new Python (`.py`) source file MUST begin with the following copyright diff --git a/CHANGELOG.md b/CHANGELOG.md index 40aa2e88..2534123b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,8 @@ automatically reconstructed into typed model instances without requiring an expl > Drain affected instances on the previous SDK, or keep their task hub on the > previous SDK and use a separate task hub for new instances. Pin or lock the core > `durabletask` dependency, including when using the Azure Managed provider. +> See [serialization and replay](docs/serialization.md) for supported types and +> compatibility guidance. ## v1.13.1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ebba57fb..8fe6f388 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -13,3 +13,31 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. + +## Serialization compatibility + +Changes to default payload serialization can alter the replay of existing +orchestrations. The [serialization guide](docs/serialization.md) describes the +policy and the current `JsonDataConverter` support matrix. Apply the same replay +assessment to provider adapters, while using each provider's own documented +serialization contract. + +For non-major releases: + +- Preserve existing encoded payloads, reconstruction, raw fallback, and failure + behavior except as described below. Defer consideration of new types and + previously unsupported cases until a breaking major release instead of adding + feature-specific opt-in flags. +- Replay-breaking changes are limited to security fixes and repairs to + previously documented supported behavior. A related type or a raw fallback + alone does not establish support. A change from failure to success can also + affect replay. +- For an allowed change that can affect replay, explain the exception and impact + in the pull request, add focused regression coverage including replay where + appropriate, and provide a changelog warning with actionable upgrade + guidance. Prefer a compatibility mitigation when feasible and safe. + +Use the pull request checklist to identify changes to payloads, reconstructed +values, or failures and describe any replay impact and applicable exception. +Applications can use their own `to_json()` / `from_json()` hooks or a custom +converter rather than expanding the default converter. diff --git a/README.md b/README.md index fb947a98..75aaf304 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ The Azure Functions Durable provider requires Python 3.13 or later. - [Supported Patterns](./docs/supported-patterns.md) - [Available Features](./docs/features.md) +- [Serialization and Replay](./docs/serialization.md) - [Getting Started](./docs/getting-started.md) - [Development Guide](./docs/development.md) - [Azure Functions Durable 2.x](./azure-functions-durable/README.md) @@ -67,6 +68,9 @@ Azure Functions uses its own shared codec rather than this converter. > replay behavior. Iteration over reconstructed sets is not replay-safe; sort > values explicitly before using their order to schedule durable work. +See [serialization and replay](./docs/serialization.md) for supported types, +custom converters, and compatibility guidance. + ### Entity IDs `EntityInstanceId` values are immutable, hashable, and usable as dictionary keys diff --git a/docs/serialization.md b/docs/serialization.md new file mode 100644 index 00000000..ebb96eab --- /dev/null +++ b/docs/serialization.md @@ -0,0 +1,158 @@ +# Serialization and replay + +The core SDK and Azure Managed provider use `JsonDataConverter` by default for +durable payloads. This guide describes that converter's current behavior and the +compatibility policy for changes to default serialization. The support matrix +reflects the current repository code; check the [changelog](../CHANGELOG.md) +for changes not yet present in an installed release. + +Azure Functions Durable uses the shared Azure Functions codec instead. Its +serialization behavior is not described by the type matrix below; see the +[provider documentation](../azure-functions-durable/README.md#unicode-json-serialization). +The compatibility policy also guides changes in this repository to that +provider's default serialization. + +## Compatibility policy + +Orchestrations can replay payloads written by an earlier SDK version. Changing +the serialized representation, reconstructed value or type, raw fallback, or +failure behavior can change orchestration decisions during replay. + +The project follows these rules for changes to default serialization: + +- New types and previously unsupported serialization or reconstruction cases + are considered for the next breaking major version. They are not added in a + non-major release, including behind a new feature-specific opt-in flag. +- Between major releases, existing payload representations, reconstructed + values and types, fallbacks, and failures are preserved except for security + fixes and repairs to previously documented supported behavior. Changes that + can affect replay include a release warning and upgrade guidance. +- Existing options, including the default-off generic set reconstruction + option, retain their current defaults between major versions, subject to the + same exceptions. + +A type is not supported merely because it is related to a supported type or +because the converter returns a raw value for it. That raw fallback can still +affect replay. A previous failure can also affect replay if application code +catches it; a change from failure to success is not automatically replay-safe. + +> [!WARNING] +> Changing annotations, hooks, converters, or their configuration can also +> change replay behavior. Test representative histories before deployment. If +> old behavior cannot be preserved, drain affected instances on the old +> configuration or isolate new instances in a separate task hub. + +## JSON representation + +The default converter serializes JSON-compatible values, dataclasses, +`SimpleNamespace` objects, and enums with JSON-compatible values. Tuples become +JSON arrays. Sets and frozensets also become arrays, ordered by canonical JSON +representation when their elements can be deterministically encoded. Custom +objects can provide a `to_json()` hook, as described below. A top-level `None` +represents no payload. + +The converter does not add or infer a Python destination type from the JSON +payload. Without a target type, decoding normally returns JSON values such as +dictionaries and lists. For compatibility, the decoder still recognizes legacy +object markers. + +## Reconstruction support + +`deserialize()` and `coerce()` accept an explicit target type, and dataclass +fields are reconstructed using their annotations. For orchestrator, activity, +and entity inputs, and for inferred activity results when a callable is +provided, the SDK first asks `can_reconstruct()` whether to use the annotation. +Explicit target types do not depend on that discovery check. + +In the table, `Sequence`, `Mapping`, and `Set` refer to `collections.abc` +types; equivalent `typing` aliases with the same origin are also recognized. + +| Target annotation | Reconstruction behavior | +| --- | --- | +| Dataclass or `from_json()` class | Rebuilt when supplied; automatically discoverable. | +| `list[T]`, `Sequence[T]` | Elements are coerced; discovery requires reconstructable `T`. | +| `dict[str, T]`, `Mapping[str, T]` | Values are coerced; discovery requires reconstructable `T`. | +| `tuple[T, ...]`, `tuple[T1, T2]` | Coerces elements; discovery needs a reconstructable element. | +| `Optional[T]` | Non-`None` values use `T`; discovery follows `T`. | +| `set[T]`, `frozenset[T]`, `Set[T]` | Generic reconstruction is opt-in; see below. | + +JSON object keys remain strings; mapping value annotations do not coerce keys. +The converter recognizes the listed container origins, not arbitrary related +ABCs or subclasses. Other generic annotations are generally passed through as +parsed JSON rather than recursively reconstructed. `T | None` is equivalent to +`Optional[T]`. + +Primitive-only annotations do not normally trigger automatic discovery. For +example, `tuple[int, ...]` is not inferred as an activity input type, so the +input remains a list; explicitly requesting that target type reconstructs a +tuple. Enums can similarly be reconstructed when explicitly requested or used +in a typed dataclass field, but an enum annotation alone is not automatically +discovered. For unions with multiple non-`None` members, the converter does not +guess which member should reconstruct an unmatched value. + +Reconstruction is best-effort: if coercion fails, `JsonDataConverter` returns +the raw value and logs the failure at debug level. A custom converter can choose +stricter behavior. + +### Set reconstruction + +Generic `set[T]`, `frozenset[T]`, and `Set[T]` annotations retain JSON arrays as +lists by default, preserving their recorded order and duplicates. Set +reconstruction for supported, hashable elements is enabled with +`JsonDataConverter(enable_generic_sets=True)`; `enable_set_coercion=True` is an +alias. Existing bare `set` and `frozenset` target types, including dataclass +fields, can still be coerced through their constructors without this option. + +> [!WARNING] +> Enabling generic set reconstruction for existing instances can change replay +> behavior. Set iteration order is not replay-safe; order values by a +> deterministic key before using their order to schedule durable work. + +## Custom types and converters + +For an application-owned type, define `to_json()` and `from_json()` hooks. The +encoder calls `to_json()` for custom values not already handled by Python's +JSON encoder; the hook returns a JSON-compatible value, not a JSON document. +Define `from_json()` as a class method or static method. The decoder calls it +with the parsed value when the type is supplied or discovered: + +```python +from durabletask.serialization import JsonDataConverter + + +class Money: + def __init__(self, cents: int) -> None: + self.cents = cents + + def to_json(self) -> dict[str, int]: + return {"cents": self.cents} + + @classmethod + def from_json(cls, value: dict[str, int]) -> "Money": + return cls(value["cents"]) + + +converter = JsonDataConverter() +restored = converter.deserialize(converter.serialize(Money(125)), Money) +assert restored.cents == 125 +``` + +The `from_json()` hook is responsible for its own nested values. It can accept +the active converter as a second parameter named `converter` and use its public +`coerce()` method to reconstruct them. + +For other cases, implement a `DataConverter` or subclass `JsonDataConverter` and +delegate unhandled values to its public methods. Implement matching +`deserialize()` and `coerce()` behavior, and override `can_reconstruct()` when +the SDK should discover your additional annotated types. A custom converter +must traverse any additional container types itself: changing discovery alone +does not add reconstruction. The +[custom converter example](../examples/custom_data_converter/README.md) shows +delegation and explains its top-level generic limitation. + +Where appropriate, an annotation listed in the support table can be used +instead. Changing existing annotations can affect replay, so apply the upgrade +precautions above. + +Keep custom serialization deterministic and configure compatible converters on +both clients and workers that exchange durable payloads. diff --git a/docs/supported-patterns.md b/docs/supported-patterns.md index b67df626..775b4d10 100644 --- a/docs/supported-patterns.md +++ b/docs/supported-patterns.md @@ -174,12 +174,13 @@ def purchase_order_workflow(ctx: task.OrchestrationContext, order: Order): return f"Approved by '{approval_details.approver}'" ``` -As an aside, you'll also notice that the example orchestration above works with custom business -objects. Custom classes, data classes, and named tuples are serialized to plain JSON automatically. -To reconstruct the original type on the receiving side, supply the type — for example via the -`data_type` argument to `wait_for_external_event` (shown above), the `return_type` argument to -`call_activity` / `call_sub_orchestrator` / `call_entity`, or by annotating the consuming function's -input parameter. Without a type, the payload is returned as plain JSON (a `dict` or `list`). +The example also works with custom business objects. Dataclasses, named tuples, and objects with +`to_json()` hooks can be serialized as plain JSON. To reconstruct a supported type on the receiving +side, supply the type — for example via the `data_type` argument to `wait_for_external_event` +(shown above), the `return_type` argument to `call_activity` / `call_sub_orchestrator` / +`call_entity`, or a supported annotation on the consuming function's input parameter. Without a +type, the payload is normally returned as plain JSON (a `dict` or `list`). See +[serialization and replay](serialization.md) for supported types and compatibility guidance. See the full [human interaction sample](../examples/human_interaction.py).