Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,35 @@ command-level Compose examples in the
[Official Image + Compose](https://github.com/durable-workflow/server/blob/main/docs/server-reference.md#official-image--compose)
section.

### Upgrading to Server 3.9.1 {#native-schema-upgrade-3-9}

Upgrading from native 3.8.x or earlier requires a schema upgrade on every
backend. Server 3.9.1 adds the empty `dw_command_intents` table:

| Backend | Schema transition from 3.8.x |
|---|---|
| SQLite | 12 → 13 |
| PostgreSQL | 13 → 14 |
| MariaDB | 13 → 14 |
| MySQL | 14 → 15 |

Stop every Server replica, keep the database running, back up the database and
Server storage, then select and pull the 3.9.1 image. Run that image's
`schema-upgrade` once before restarting every replica on the same version.
Server does not upgrade an existing schema at startup: it refuses an old
schema with `native_schema_upgrade_required`. Verify readiness, reported
version and worker registration after restart. The
[qualified 3.9.1 release instructions](https://github.com/durable-workflow/server/releases/tag/3.9.1#upgrade)
provide the source-free Compose commands.

MySQL and MariaDB upgrades resume after interruption; SQLite and PostgreSQL
apply the step transactionally. Rollback to 3.8.x requires restoring the backup:
3.8.0 refuses the upgraded database with `native_schema_newer_than_binary`.
From 3.9.0, no further schema step is needed. From older versions,
`schema-upgrade` applies the earlier steps too, including the
[3.7 JSON rebuild](#native-schema-upgrade-3-7) where applicable. Worker protocol
1.20 and control plane 2 are unchanged.

### Upgrading to Server 3.8.0 {#native-schema-upgrade-3-8}

Server 3.8.0 needs a native schema upgrade on every backend. It adds one
Expand Down
9 changes: 7 additions & 2 deletions docs/polyglot/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -461,15 +461,20 @@ end-to-end rollout walkthrough; the CLI mirrors of these methods are
| `await client.describe_workflow_run(workflow_id, run_id)` | `WorkflowRun` | `WorkflowNotFound`, auth/server errors |
| `await client.get_history(workflow_id, run_id)` | decoded history payload | `WorkflowNotFound`, auth/server errors |
| `await client.export_history(workflow_id, run_id)` | decoded archival history payload | `WorkflowNotFound`, auth/server errors |
| `await client.signal_workflow(workflow_id, signal_name, args=None)` | `None` | `WorkflowNotFound`, `InvalidArgument`, auth/server errors |
| `await client.query_workflow(workflow_id, query_name, args=None)` | decoded query result | `QueryFailed`, `WorkflowNotFound`, auth/server errors |
| `await client.signal_workflow(workflow_id, signal_name, args=None)` | admission receipt (`dict`) | `WorkflowNotFound`, `InvalidArgument`, auth/server errors |
| `await client.query_workflow(workflow_id, query_name, args=None)` | query receipt (`dict`) | `QueryFailed`, `WorkflowNotFound`, auth/server errors |
| `await client.update_workflow(workflow_id, update_name, args=None, ...)` | update receipt (`dict`) | `UpdateRejected`, `UpdateValidationFailed`, `UpdateFailed`, `WorkflowNotFound`, auth/server errors |
| `await client.cancel_workflow(workflow_id, reason=None)` | `None` | `WorkflowNotFound`, auth/server errors |
| `await client.terminate_workflow(workflow_id, reason=None)` | `None` | `WorkflowNotFound`, auth/server errors |
| `await client.repair_workflow(workflow_id)` | `WorkflowCommandResult` | `WorkflowNotFound`, auth/server errors |
| `await client.archive_workflow(workflow_id, reason=None)` | `WorkflowCommandResult` | `WorkflowNotFound`, auth/server errors |
| `await client.get_result(handle, poll_interval=0.5, timeout=30.0)` | decoded workflow output | `WorkflowFailed`, `WorkflowCancelled`, `WorkflowTerminated`, `TimeoutError` |

`signal_workflow` returns the admission receipt, including its `request_id`.
`query_workflow` and `WorkflowHandle.query` return a query receipt; read
`receipt["result"]` for the decoded handler value, including when the handler successfully
returns `None`. See the [Python client guide](https://python.durable-workflow.com/reference/client/).

`update_workflow` defaults to `wait_for="accepted"`. Its receipt can have
`update_status="accepted"` and `result=None`. With `wait_for="completed"`,
`wait_timeout_seconds` bounds the wait. On expiry, the receipt stays accepted
Expand Down
20 changes: 16 additions & 4 deletions docs/polyglot/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,9 +148,11 @@ is `Ok(Value::Null)`. Receipt reads preserve the explicit `update_status`.
See the [Client API](https://rust.durable-workflow.com/durable_workflow/struct.Client.html#method.update_outcome)
for the Rust interface.

Rust SDK 4.15.0 still applies the ordinary payload ceiling to the whole
command batch. See the [known planner limit](./server-config-reference.md#rust-completion-planning)
before running a large fan-out without external payload storage.
Rust SDK 4.16 uses the Server's explicit
`limits.max_workflow_task_completion_bytes` budget for completion planning.
Older producers without that field keep the conservative ordinary cap. See
[completion planning](./server-config-reference.md#rust-completion-planning)
for the qualified native 3.9.1 tuple and payload-placement limits.

## Condition wait deadlines and worker upgrades {#condition-wait-deadlines}

Expand Down Expand Up @@ -235,12 +237,22 @@ Each observed client request, worker poll or claimed-task dispatch increments on
| `durable_workflow_worker_polls` | `durable_workflow_worker_poll_duration_seconds` |
| `durable_workflow_worker_tasks` | `durable_workflow_worker_task_duration_seconds` |

Client tag keys are exactly `method`, `route`, `plane`, `status_code`, `outcome`; worker keys are `task_kind`, `task_queue`, `outcome`. Reviewed route placeholders omit IDs, query strings, payloads and credentials; unknown paths use `unknown`. Queue names are application-controlled: use a bounded set, not one queue per run.
For the counter/duration pairs above, client tag keys are exactly `method`, `route`, `plane`, `status_code`, `outcome`; worker keys are `task_kind`, `task_queue`, `outcome`. Reviewed route placeholders omit IDs, query strings, payloads and credentials; unknown paths use `unknown`. Queue names are application-controlled: use a bounded set, not one queue per run.

Client outcomes include `ok`, `http_error`, `bridge_rejected`, `TransportError`, `TimeoutError`, `SdkError` and `pending` for a cancelled future. `status_code` keeps an observed decimal HTTP status, otherwise `none`. Rust exception outcome names need not match another SDK’s names.

One client observation includes internal transport/storage-admission retries and response/payload handling; application/worker-managed calls are separate observations. A lazy iterator observes each fetched page. Worker polls include managed retry/backoff and distinguish `task`, `empty`, `backpressure`, `error`. Task outcomes `completed`, `failed`, `expired` describe dispatch processing, not the workflow’s terminal state. These are process observations, not durable counts, billing units or schedule-to-start latency; abrupt process loss can omit a final sample.

Rust SDK 4.16 adds the seventh instrument,
`durable_workflow_worker_task_schedule_to_start_seconds`, with only
`task_kind` and `task_queue` tags. It records Server-measured eligibility-to-claim
delay in seconds when the producer advertises `claimed_task_ready_age` and
supplies valid `ready_age_ms`. Missing, invalid or unsupported data produces no
sample; the SDK never invents a zero. The
[published 4.16/native 3.9.1 controls](https://github.com/durable-workflow/sdk-rust/pull/258#issuecomment-6854288671)
(linux/amd64, SQLite) qualify all seven metrics, workflow/activity ready-age
observations and the Prometheus adapter.

For the official Prometheus adapter, enable the optional `metrics-prometheus` Cargo feature. This example uses the exact released dependency recipe:

```toml
Expand Down
78 changes: 53 additions & 25 deletions docs/polyglot/server-config-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,12 @@ listed in its [implementation notes](#native-server-3x). On native 3.8.0,
workflow-task completions, rejecting nonempty bodies. Its operation-name
ceiling is fixed at 256 bytes.

Native 3.9.1 still applies a 255-byte UTF-8 check to signal `request_id`.
A 255-character multibyte key can therefore return `422 invalid_request_id`.
The [installed CLI qualification](https://github.com/durable-workflow/server/issues/934#issuecomment-6853943056)
tracks this published limitation; it does not establish full request-key
acceptance under shared #216.

The currently-configured values are published under `limits` on the `GET /api/cluster/info` response, so clients can discover them at runtime:

```json
Expand Down Expand Up @@ -403,7 +409,7 @@ limit; child-call fields have stricter bounds than other commands.
| `workflow_type` | 255 | `continue_as_new` |
| `workflow_type` / `queue` | 191 | `start_child_workflow` |

Native 3.8.0 and the corrected source also retain UTF-8 byte checks.
Native 3.8.0 and 3.9.1 also retain UTF-8 byte checks.
`activity_type`, `open_signal_wait.signal_name`, an explicit
`schedule_activity.queue`, and `continue_as_new.workflow_type` / `queue`
use a 255-byte parser ceiling (`422 invalid_request`). Child `workflow_type`
Expand All @@ -412,15 +418,14 @@ and `queue` have an additional 191-byte guard that can return
128 copies of `é` is 256 bytes and is refused despite having only 128
characters. The new character checks do not remove these byte guards.

Native Server 3.8.0 does not consistently validate these command-name bounds
before writes. SQLite can silently store over-long names; other backends can
return database errors. The [source correction](https://github.com/durable-workflow/server/pull/1186)
adds field-specific `422` refusals before writes, such as
`commands.0.activity_type`, but awaits a qualified published Server release.
The selected stable remains 3.8.0. Review existing activity types and signal
waits longer than 191 characters before upgrading to that corrected release:
new commands using those names will be refused. The remaining identifier
sweep is tracked in [Server #607](https://github.com/durable-workflow/server/issues/607).
Native Server 3.9.1 includes the [published correction](https://github.com/durable-workflow/server/releases/tag/3.9.1#other-changes):
field-specific `422 validation_failed` refusals, such as
`commands.0.activity_type`, before writes. Native 3.8.0 did not consistently
validate these character bounds; SQLite could silently store over-long names
while other backends could return database errors. Review existing activity
types and signal waits longer than 191 characters before upgrading: new
commands using those names are now refused. The remaining identifier sweep is
tracked in [Server #607](https://github.com/durable-workflow/server/issues/607).
The 256-byte HTTP operation-name ceiling above is a separate boundary.

### Workflow-task completion size {#workflow-task-completion-size}
Expand All @@ -434,6 +439,19 @@ still meet the per-payload limit inside the completion. PHP Server 2.x and
earlier native releases cap a completion at one payload limit, like any other
request.

Native 3.9.1 advertises the actual budget as
`limits.max_workflow_task_completion_bytes` in cluster info. It measures the
complete encoded JSON request body, including envelopes, metadata and commands,
for both completion routes. Use that advertised value rather than inferring a
multiplier from the Server version or summing decoded arguments. Per-value and
ordinary-route limits remain unchanged. The field grants no additional external
payload upload or reference-placement authority.

Rust SDK 4.16 completion planning honors a positive advertised cap even below
the ordinary limit. An absent field preserves the conservative fallback; an explicitly
malformed, null, zero, negative or non-integer value refuses completion planning
before upload. Ordinary routes ignore this unused completion field.

A larger completion is refused with `413 payload_too_large` and
`limit_kind: workflow_task_completion_bytes` before anything is recorded.
Replaying the task produces the same commands, so sending the completion again
Expand All @@ -444,29 +462,39 @@ sizes, and the Server records the failure with
`failure_category = structural_limit`.

Python SDK 2.17 and Rust SDK 4.15 include handling for the typed refusal.
On Server 3.8.0, an oversized completion can fail at the transport before the
worker receives HTTP 413. In the published Python test, `httpx.ReadError` left
the workflow pending instead of failing structurally. Published-Server
over-limit acceptance remains open under the
[Server response fix](https://github.com/durable-workflow/.github/issues/228#issuecomment-6848969678).
Reduce completion sizes until that path is qualified.
On Server 3.8.0, a published Python control hit `httpx.ReadError` before receiving
HTTP 413 and left the workflow pending. Native 3.9.1 corrects that response
path with bounded request discard: at most 64 MiB of remaining upload data for
at most two seconds before returning the original refusal and closing the
connection. Uploads beyond either bound may still encounter transport errors.
Published Python SDK 2.17.1 with qualified native Server 3.9.1
(linux/amd64, SQLite) now passes the
[actual installed-worker controls](https://github.com/durable-workflow/.github/issues/228#issuecomment-6853834512):
300 × 10 KB within-cap fan-out, one typed 413 and one structural failure with
zero partial activities for an oversized completion, then a successful task on
the same worker. This qualifies that exact Python/Server tuple. PHP requires its own installed
acceptance; Rust 4.16 is qualified below. Shared #228 remains open. Request-key
acceptance under shared #216 is separate.

To keep a completion within the limit:

- Split a large fan-out into smaller batches across workflow tasks.
- Move work into child workflows, or continue as new.
- Store large arguments with external payload storage, so the completion
carries references instead of the payloads.
- Use external payload storage only for supported payload placements.
Activity-continuation command arguments remain inline; an activity/result
upload allowance does not authorize references in those command arguments.

#### Rust command planning {#rust-completion-planning}

Rust SDK 4.15.0 still uses the ordinary per-payload limit as the aggregate
command-planning ceiling (2 MiB by default), before the Server's larger
completion allowance. Without namespace external payload storage, the
published 300 × 10 KB fan-out is refused with `external_payload_unavailable`;
the worker exits and the run remains pending. Use smaller command batches
while the [SDK aggregate planner fix](https://github.com/durable-workflow/sdk-rust/issues/254)
is pending. This SDK path does not yet implement the larger Server allowance.
Rust SDK 4.16 uses the positive explicit completion budget advertised by the
producer. The [published 4.16/native 3.9.1 controls](https://github.com/durable-workflow/sdk-rust/pull/258#issuecomment-6854288671)
(linux/amd64, SQLite) pass 300 × 10 KB inline fan-out without external storage,
an oversized 60 × 450 KB batch failing structurally with zero partial activities,
and a successful follow-up on the same worker. Without the explicit field,
older producers keep the conservative ordinary cap. The larger budget grants
no new upload or reference-placement authority; activity-continuation command
arguments remain inline. These controls do not qualify request-key retries
under shared #216 or native timeout cleanup.

## Docker Bootstrap And Provenance

Expand Down
2 changes: 2 additions & 0 deletions docs/polyglot/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,8 @@ curl -H "Authorization: Bearer $DW_AUTH_TOKEN" \
The stack is the native Server and MySQL. Compose refuses to start until
`DW_AUTH_TOKEN` is set. The [deployment guide](/docs/deployment#single-node-production)
covers the production env file, backups and upgrades.
Upgrades from native 3.8.x or earlier to 3.9.1 require the
[explicit schema upgrade](../deployment.md#native-schema-upgrade-3-9) before restart.

### Ports

Expand Down
15 changes: 15 additions & 0 deletions i18n/de/docusaurus-plugin-content-docs/current/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,21 @@ Upgrade-Reihenfolge:

Der native Server prüft sein Schema beim Start. Patch-Veröffentlichungen innerhalb einer nativen Minor-Linie behalten das Schema bei, daher genügt ein Image-Wechsel. Eine Veröffentlichung, deren Hinweise eine Schemaänderung ankündigen, benötigt zusätzlich den Befehl `schema-upgrade` des Images, ausgeführt nach einer Datenbanksicherung, während alle Server gestoppt sind. Die Server-Referenz enthält aktuelle Compose-Befehlsbeispiele unter [Offizielles Image + Compose](https://github.com/durable-workflow/server/blob/main/docs/server-reference.md#official-image--compose).

### Upgrade auf Server 3.9.1 {#native-schema-upgrade-3-9}

Ein Upgrade von native 3.8.x oder früher erfordert auf jedem Backend ein Schema-Upgrade. Server 3.9.1 fügt die leere Tabelle `dw_command_intents` hinzu:

| Backend | Schemaübergang von 3.8.x |
|---|---|
| SQLite | 12 → 13 |
| PostgreSQL | 13 → 14 |
| MariaDB | 13 → 14 |
| MySQL | 14 → 15 |

Stoppen Sie alle Server-Replikate, lassen Sie die Datenbank laufen, sichern Sie Datenbank und Server-Speicher und wählen und laden Sie dann das Image 3.9.1. Führen Sie dessen `schema-upgrade` einmal aus, bevor alle Replikate auf derselben Version neu starten. Server aktualisiert ein vorhandenes Schema nicht beim Start: Ein altes Schema wird mit `native_schema_upgrade_required` abgelehnt. Prüfen Sie danach Readiness, gemeldete Version und Worker-Registrierung. Die [qualifizierte Release-Anleitung für 3.9.1](https://github.com/durable-workflow/server/releases/tag/3.9.1#upgrade) enthält die Compose-Befehle ohne Quellcode-Checkout.

MySQL und MariaDB setzen unterbrochene Upgrades fort; SQLite und PostgreSQL führen den Schritt transaktional aus. Ein Rollback auf 3.8.x erfordert das Backup: 3.8.0 lehnt die aktualisierte Datenbank mit `native_schema_newer_than_binary` ab. Von 3.9.0 ist kein weiterer Schemaschritt nötig. Von älteren Versionen führt `schema-upgrade` auch die vorherigen Schritte aus, gegebenenfalls einschließlich des [JSON-Neuaufbaus aus 3.7](#native-schema-upgrade-3-7). Worker-Protokoll 1.20 und Control Plane 2 bleiben unverändert.

### Upgrade auf Server 3.8.0 {#native-schema-upgrade-3-8}

Server 3.8.0 benötigt auf jedem Backend ein natives Schema-Upgrade. Es fügt eine nur native Tabelle hinzu, `dw_task_queue_pauses`, die leer angelegt wird:
Expand Down
Loading
Loading