|
| 1 | +# Serialization and replay |
| 2 | + |
| 3 | +The core SDK and Azure Managed provider use `JsonDataConverter` by default for |
| 4 | +durable payloads. This guide describes that converter's current behavior and the |
| 5 | +compatibility policy for changes to default serialization. The support matrix |
| 6 | +reflects the current repository code; check the [changelog](../CHANGELOG.md) |
| 7 | +for changes not yet present in an installed release. |
| 8 | + |
| 9 | +Azure Functions Durable uses the shared Azure Functions codec instead. Its |
| 10 | +serialization behavior is not described by the type matrix below; see the |
| 11 | +[provider documentation](../azure-functions-durable/README.md#unicode-json-serialization). |
| 12 | +The compatibility policy also guides changes in this repository to that |
| 13 | +provider's default serialization. |
| 14 | + |
| 15 | +## Compatibility policy |
| 16 | + |
| 17 | +Orchestrations can replay payloads written by an earlier SDK version. Changing |
| 18 | +the serialized representation, reconstructed value or type, raw fallback, or |
| 19 | +failure behavior can change orchestration decisions during replay. |
| 20 | + |
| 21 | +The project follows these rules for changes to default serialization: |
| 22 | + |
| 23 | +- New types and previously unsupported serialization or reconstruction cases |
| 24 | + are considered for the next breaking major version. They are not added in a |
| 25 | + non-major release, including behind a new feature-specific opt-in flag. |
| 26 | +- Between major releases, existing payload representations, reconstructed |
| 27 | + values and types, fallbacks, and failures are preserved except for security |
| 28 | + fixes and repairs to previously documented supported behavior. Changes that |
| 29 | + can affect replay include a release warning and upgrade guidance. |
| 30 | +- Existing options, including the default-off generic set reconstruction |
| 31 | + option, retain their current defaults between major versions, subject to the |
| 32 | + same exceptions. |
| 33 | + |
| 34 | +A type is not supported merely because it is related to a supported type or |
| 35 | +because the converter returns a raw value for it. That raw fallback can still |
| 36 | +affect replay. A previous failure can also affect replay if application code |
| 37 | +catches it; a change from failure to success is not automatically replay-safe. |
| 38 | + |
| 39 | +> [!WARNING] |
| 40 | +> Changing annotations, hooks, converters, or their configuration can also |
| 41 | +> change replay behavior. Test representative histories before deployment. If |
| 42 | +> old behavior cannot be preserved, drain affected instances on the old |
| 43 | +> configuration or isolate new instances in a separate task hub. |
| 44 | +
|
| 45 | +## JSON representation |
| 46 | + |
| 47 | +The default converter serializes JSON-compatible values, dataclasses, |
| 48 | +`SimpleNamespace` objects, and enums with JSON-compatible values. Tuples become |
| 49 | +JSON arrays. Sets and frozensets also become arrays, ordered by canonical JSON |
| 50 | +representation when their elements can be deterministically encoded. Custom |
| 51 | +objects can provide a `to_json()` hook, as described below. A top-level `None` |
| 52 | +represents no payload. |
| 53 | + |
| 54 | +The converter does not add or infer a Python destination type from the JSON |
| 55 | +payload. Without a target type, decoding normally returns JSON values such as |
| 56 | +dictionaries and lists. For compatibility, the decoder still recognizes legacy |
| 57 | +object markers. |
| 58 | + |
| 59 | +## Reconstruction support |
| 60 | + |
| 61 | +`deserialize()` and `coerce()` accept an explicit target type, and dataclass |
| 62 | +fields are reconstructed using their annotations. For orchestrator, activity, |
| 63 | +and entity inputs, and for inferred activity results when a callable is |
| 64 | +provided, the SDK first asks `can_reconstruct()` whether to use the annotation. |
| 65 | +Explicit target types do not depend on that discovery check. |
| 66 | + |
| 67 | +In the table, `Sequence`, `Mapping`, and `Set` refer to `collections.abc` |
| 68 | +types; equivalent `typing` aliases with the same origin are also recognized. |
| 69 | + |
| 70 | +| Target annotation | Reconstruction behavior | |
| 71 | +| --- | --- | |
| 72 | +| Dataclass or `from_json()` class | Rebuilt when supplied; automatically discoverable. | |
| 73 | +| `list[T]`, `Sequence[T]` | Elements are coerced; discovery requires reconstructable `T`. | |
| 74 | +| `dict[str, T]`, `Mapping[str, T]` | Values are coerced; discovery requires reconstructable `T`. | |
| 75 | +| `tuple[T, ...]`, `tuple[T1, T2]` | Coerces elements; discovery needs a reconstructable element. | |
| 76 | +| `Optional[T]` | Non-`None` values use `T`; discovery follows `T`. | |
| 77 | +| `set[T]`, `frozenset[T]`, `Set[T]` | Generic reconstruction is opt-in; see below. | |
| 78 | +| `MutableMapping[str, T]`, `MutableSequence[T]` | Not discovered or traversed; values stay raw. | |
| 79 | + |
| 80 | +JSON object keys remain strings; mapping value annotations do not coerce keys. |
| 81 | +The converter recognizes the listed container origins, not arbitrary related |
| 82 | +ABCs or subclasses. Other generic annotations are generally passed through as |
| 83 | +parsed JSON rather than recursively reconstructed. `T | None` is equivalent to |
| 84 | +`Optional[T]`. |
| 85 | + |
| 86 | +Primitive-only annotations do not normally trigger automatic discovery. For |
| 87 | +example, `tuple[int, ...]` is not inferred as an activity input type, so the |
| 88 | +input remains a list; explicitly requesting that target type reconstructs a |
| 89 | +tuple. Enums can similarly be reconstructed when explicitly requested or used |
| 90 | +in a typed dataclass field, but an enum annotation alone is not automatically |
| 91 | +discovered. For unions with multiple non-`None` members, the converter does not |
| 92 | +guess which member should reconstruct an unmatched value. |
| 93 | + |
| 94 | +Reconstruction is best-effort: if coercion fails, `JsonDataConverter` returns |
| 95 | +the raw value and logs the failure at debug level. A custom converter can choose |
| 96 | +stricter behavior. |
| 97 | + |
| 98 | +### Set reconstruction |
| 99 | + |
| 100 | +Generic `set[T]`, `frozenset[T]`, and `Set[T]` annotations retain JSON arrays as |
| 101 | +lists by default, preserving their recorded order and duplicates. Set |
| 102 | +reconstruction for supported, hashable elements is enabled with |
| 103 | +`JsonDataConverter(enable_generic_sets=True)`; `enable_set_coercion=True` is an |
| 104 | +alias. Existing bare `set` and `frozenset` target types, including dataclass |
| 105 | +fields, can still be coerced through their constructors without this option. |
| 106 | + |
| 107 | +> [!WARNING] |
| 108 | +> Enabling generic set reconstruction for existing instances can change replay |
| 109 | +> behavior. Set iteration order is not replay-safe; order values by a |
| 110 | +> deterministic key before using their order to schedule durable work. |
| 111 | +
|
| 112 | +## Custom types and converters |
| 113 | + |
| 114 | +For an application-owned type, define `to_json()` and `from_json()` hooks. The |
| 115 | +encoder calls `to_json()` for custom values not already handled by Python's |
| 116 | +JSON encoder; the hook returns a JSON-compatible value, not a JSON document. |
| 117 | +Define `from_json()` as a class method or static method. The decoder calls it |
| 118 | +with the parsed value when the type is supplied or discovered: |
| 119 | + |
| 120 | +```python |
| 121 | +from durabletask.serialization import JsonDataConverter |
| 122 | + |
| 123 | + |
| 124 | +class Money: |
| 125 | + def __init__(self, cents: int) -> None: |
| 126 | + self.cents = cents |
| 127 | + |
| 128 | + def to_json(self) -> dict[str, int]: |
| 129 | + return {"cents": self.cents} |
| 130 | + |
| 131 | + @classmethod |
| 132 | + def from_json(cls, value: dict[str, int]) -> "Money": |
| 133 | + return cls(value["cents"]) |
| 134 | + |
| 135 | + |
| 136 | +converter = JsonDataConverter() |
| 137 | +restored = converter.deserialize(converter.serialize(Money(125)), Money) |
| 138 | +assert restored.cents == 125 |
| 139 | +``` |
| 140 | + |
| 141 | +The `from_json()` hook is responsible for its own nested values. It can accept |
| 142 | +the active converter as a second parameter named `converter` and use its public |
| 143 | +`coerce()` method to reconstruct them. |
| 144 | + |
| 145 | +For other cases, implement a `DataConverter` or subclass `JsonDataConverter` and |
| 146 | +delegate unhandled values to its public methods. Implement matching |
| 147 | +`deserialize()` and `coerce()` behavior, and override `can_reconstruct()` when |
| 148 | +the SDK should discover your additional annotated types. A custom converter |
| 149 | +must traverse any additional container types itself: changing discovery alone |
| 150 | +does not add reconstruction. The |
| 151 | +[custom converter example](../examples/custom_data_converter/README.md) shows |
| 152 | +delegation and explains its top-level generic limitation. |
| 153 | + |
| 154 | +For example, where appropriate, `dict[str, Order]` and `list[Order]` annotations |
| 155 | +can be used instead of `MutableMapping[str, Order]` and |
| 156 | +`MutableSequence[Order]`. Changing existing annotations can affect replay, so |
| 157 | +apply the upgrade precautions above. |
| 158 | + |
| 159 | +Keep custom serialization deterministic and configure compatible converters on |
| 160 | +both clients and workers that exchange durable payloads. |
0 commit comments