diff --git a/docs/production-deployment/self-hosted-guide/archival.mdx b/docs/production-deployment/self-hosted-guide/archival.mdx index b437648ca7..0d8742604b 100644 --- a/docs/production-deployment/self-hosted-guide/archival.mdx +++ b/docs/production-deployment/self-hosted-guide/archival.mdx @@ -145,8 +145,18 @@ supports only a single Archival URI, but each Namespace can use a different URI. between `enabled` and `disabled` states as long as Archival is enabled at the Temporal Service level. Archival is supported in [Global Namespaces](/global-namespace) (Namespaces that span multiple clusters). When Archival -is running in a Global Namespace, it first runs on the active cluster; later it runs on the standby cluster. Before -archiving, a history check is done to see what has been previously archived. +is running in a Global Namespace, each Temporal Service independently creates and processes Archival tasks. When all +participating Temporal Services use the same Archival URI, the first one writes the history blob and the others check +that it already exists. This produces one stored copy, but every Temporal Service sends requests to the Archival +backend. + +Monitor `history_archiver_blob_exists`, `history_archiver_archive_transient_error`, and +`visibility_archiver_archive_transient_error` on every Temporal Service. If a shared backend is throttled, use +`history.archivalBackendMaxRPS` to limit request rates or `history.archivalProcessorArchiveDelay` to spread Archival +tasks over a longer period. These are dynamic configuration settings and do not require a restart. + +Do not try to disable Archival only on a passive Temporal Service. Archival configuration is part of the Namespace +metadata and replicates to the other Temporal Services. #### Test your Archival setup {/* #test-your-archival-setup */} diff --git a/docs/production-deployment/self-hosted-guide/multi-cluster-replication.mdx b/docs/production-deployment/self-hosted-guide/multi-cluster-replication.mdx index 12759772c0..ab394e513d 100644 --- a/docs/production-deployment/self-hosted-guide/multi-cluster-replication.mdx +++ b/docs/production-deployment/self-hosted-guide/multi-cluster-replication.mdx @@ -26,9 +26,100 @@ As a result, [Temporal UI](/web-ui) works with Global Namespaces the same way it Applications making API calls directly to the Temporal Visibility API continue to work even if a Global Namespace is in standby mode. However, they might see a lag due to replication delay when querying the Workflow Execution state from a standby Cluster. +## How to prepare for a Namespace failover + +Prepare the replication topology and monitor the destination Temporal Service before you start a planned handover. + +### Configure compatible shard counts + +The History shard counts of connected Temporal Services must be equal or clean multiples of each other. For example, +512 and 1,024 shards are compatible, but 512 and 768 are not. The replication stream rejects a connection when it +cannot map History shards between the source and destination. + +Set [`numHistoryShards`](/references/configuration#numhistoryshards) when you first deploy each Temporal Service. You +cannot change it after the Temporal Service has started. + +### Configure API forwarding + +Use `all-apis-forwarding` on every participating Temporal Service if Clients and Workers can continue connecting to any +frontend after a failover: + +```yaml +dcRedirectionPolicy: + policy: "all-apis-forwarding" +``` + +This static configuration requires a Temporal Server restart. It forwards Client requests, Worker polls, Activity +heartbeats, and task-completion requests from a passive Temporal Service to the active Temporal Service. + +`selected-apis-forwarding` forwards only selected Client APIs. It does not forward Worker polls, task-completion +requests, Activity heartbeats, `UpdateWorkflowExecution`, or `ExecuteMultiOperation`. If you use that policy, repoint +Clients and Workers to the destination frontend as part of the failover procedure. + +Keep `system.enableNamespaceNotActiveAutoForwarding` at its default value of `true` on every Temporal Service. +Setting it to `false` disables forwarding regardless of `dcRedirectionPolicy`. + +### Populate a new passive Temporal Service + +The replication stream transfers tasks created after two Temporal Services are connected. If you add a passive +Temporal Service to an existing Global Namespace, run `force-replication-v2` to copy existing Workflow Execution state +before the first failover: + +```bash +temporal workflow start \ + --address ActiveClusterFrontendAddress \ + --namespace temporal-system \ + --task-queue default-worker-tq \ + --type force-replication-v2 \ + --workflow-id force-replication-YourNamespaceName \ + --input '{"Namespace":"YourNamespaceName","Query":"","ConcurrentActivityCount":5}' +``` + +Wait for this Workflow to complete before starting a handover. Do not run it after a temporary replication outage on +an established passive Temporal Service. An established stream resumes from its replication watermark when connectivity +returns. + +Set `history.EnableReplicationTaskTieredProcessing` to `true` on every Temporal Service before running a backfill or +handover. This separates live, high-priority replication tasks from lower-priority backfill tasks so a backfill does not +block the handover drain. Changing this dynamic configuration restarts replication streams; wait for them to recover +before continuing. + +### Check replication health + +Do not start a planned handover until the destination Temporal Service is caught up and the replication stream is +making progress. Monitor the following [Temporal Service metrics](/references/cluster-metrics#multi-cluster-replication-metrics): + +| Metric | Ready-to-handover condition | +| --- | --- | +| `replication_stream_stuck` | No new events. A non-zero rate means a stream stopped making progress. | +| `replication_dlq_enqueue_failed` | No new events. A failure can mean a replication task was not stored for later recovery. | +| `replication_tasks_lag` | Low and stable or decreasing on the destination. Do not proceed while lag is rising. | +| `replication_latency` | The p99 latency is comfortably below the handover timeout. Tasks near the 30-second maximum drain window cannot complete in time. | +| `replication_tasks_recv_backlog` | Low and not approaching the receiver's flow-control limit. | +| `task_errors_discarded` | No new events on the passive Temporal Service. A discarded standby task can leave a Workflow Execution without a pending task after failover. | + +A non-zero `replication_stream_error` rate does not by itself indicate that replication is unhealthy. Connections can +be recycled and re-established while the stream continues to make progress. Evaluate it together with +`replication_stream_stuck`, replication lag, and replication latency. + +Keep `history.standbyTaskMissingEventsDiscardDelay` at its default of 15 minutes or higher. A shorter delay makes a +passive Temporal Service discard standby tasks sooner when the Event History they require is delayed. Monitor +`task_errors_discarded` to detect these discards before failover. + +### Prepare Schedules + +If the Namespace uses [Schedules](/schedule), confirm that `worker.enableScheduler` is `true` on the destination +Temporal Service and that the Worker Service has enough capacity for the Schedule load. Align +`worker.schedulerNamespaceStartWorkflowRPS` between the source and destination so catchup behavior does not change +after failover. + +A Schedule's `catchup_window` must be longer than the expected handover drain. The default catchup window is much +longer than the handover's 30-second maximum drain time, but a custom short window can cause a scheduled action to be +missed during the handover. + ## How to fail over a Global Namespace -For a planned failover, run the built-in `namespace-handover` Workflow against the currently active Temporal Service. +For a planned failover, run the built-in `namespace-handover-v2` Workflow against the currently active Temporal Service. The handover Workflow: 1. Waits for the destination Temporal Service to approach the configured replication-lag thresholds. @@ -40,22 +131,79 @@ This process creates a brief availability gap while traffic is paused, but it pr back at the cutover. If the destination cannot catch up during the handover timeout, the Workflow restores the `normal` replication state without changing the active Temporal Service. -Connect Temporal CLI to the currently active Temporal Service, then run the handover Workflow in the `temporal-system` -Namespace. Use a unique Workflow Id for each handover: +First check whether a handover is already running. Never run two handovers for the same Namespace at the same time: + +```bash +temporal workflow list \ + --address ActiveClusterFrontendAddress \ + --namespace temporal-system \ + --query 'WorkflowType="namespace-handover-v2" AND ExecutionStatus="Running"' +``` + +If no handover is running for the Namespace, start the Workflow in the `temporal-system` Namespace: ```bash -temporal workflow execute \ +temporal workflow start \ --address ActiveClusterFrontendAddress \ --namespace temporal-system \ --task-queue default-worker-tq \ - --workflow-id YourHandoverWorkflowId \ - --type namespace-handover \ - --input '{"Namespace":"YourNamespaceName","RemoteCluster":"NewActiveCluster","AllowedLaggingSeconds":10,"AllowedLaggingTasks":0,"HandoverTimeoutSeconds":10}' + --workflow-id handover-YourNamespaceName \ + --type namespace-handover-v2 \ + --input '{"Namespace":"YourNamespaceName","RemoteCluster":"NewActiveCluster","AllowedLaggingSeconds":10,"AllowedLaggingTasks":500,"HandoverTimeoutSeconds":30}' +``` + +The input fields have the following behavior: + +| Field | Behavior | +| --- | --- | +| `RemoteCluster` | The configured name of the destination Temporal Service. | +| `AllowedLaggingSeconds` | The maximum time lag before the Workflow can enter the drain phase. The server limits this value to between 5 and 120 seconds. | +| `AllowedLaggingTasks` | The maximum task-count lag before the Workflow can enter the drain phase. After the destination reaches the replication snapshot taken when the handover starts, a shard passes when either its time lag or task-count lag is within the corresponding limit. A value of `0` is the strictest task-count threshold; it does not disable the check. | +| `HandoverTimeoutSeconds` | The time allowed for every shard to drain after writes pause. Set it explicitly. The maximum is 30 seconds. | + +While the Workflow is catching up, `catchup_not_ready_shard_count` should fall to zero. During the drain phase, +`handover_ready_shard_count` should reach the Temporal Service's `numHistoryShards` value before the handover timeout. +The Workflow rolls back if every shard does not become ready in time. + +The Workflow automatically restores the Namespace to the `normal` replication state if the destination cannot drain +within the timeout. Do not terminate a handover Workflow. Termination prevents Workflow cleanup logic from running and +can leave the Namespace in the `handover` replication state with write requests blocked. + +### Verify a planned failover + +After the handover Workflow completes, describe the Namespace against the source and destination frontends: + +```bash +temporal operator namespace describe \ + --address ClusterFrontendAddress \ + --namespace YourNamespaceName ``` -`RemoteCluster` is the configured name of the destination Temporal Service. `AllowedLaggingSeconds` and -`AllowedLaggingTasks` control when the Workflow can begin pausing traffic. `HandoverTimeoutSeconds` controls how long -it waits for complete replication after the pause and is capped at 30 seconds. +The source and destination Temporal Services must report the same new active Temporal Service and a replication state +of `Normal`. The Workflow completing does not replace this check because Namespace metadata replication is +asynchronous. + +Continue monitoring after the change: + +- Confirm `replication_tasks_applied` begins increasing on the former active Temporal Service. This verifies that the + reverse replication stream is active. +- Confirm `replication_tasks_lag` settles on the former active Temporal Service. +- Confirm Workflow Task Schedule-to-Start latency returns to its normal range on the new active Temporal Service. +- Confirm Clients and Workers no longer produce `NamespaceNotActive` or forwarding errors. +- If Clients or Workers still connect to the former active Temporal Service, confirm `client_redirection_requests` + increases and `client_redirection_errors` remains at zero. +- If the Namespace uses Schedules, run `temporal schedule describe` against the new active Temporal Service and confirm + that `FutureActionTimes` contains future times and `RecentActions` advances. + +If the source and destination Temporal Services disagree about the active Temporal Service, send an explicit Namespace +update to the new active frontend, and then repeat the checks against both frontends: + +```bash +temporal operator namespace update \ + --address NewActiveClusterFrontendAddress \ + --namespace YourNamespaceName \ + --active-cluster NewActiveCluster +``` ### Force a failover when the active Temporal Service is unavailable @@ -487,6 +635,95 @@ T = 2: task A is loaded. At this time, due to the rebuild of a Workflow Execution's mutable state (conflict resolution), Task A is no longer relevant (Task A's corresponding Event belongs to non-current branch). Task processing logic will verify both the Event Id and version of the Task against a corresponding Workflow Execution's mutable state, then discard task A. +## How to maintain replicated data + +### Delete Workflow Executions + +Run explicit Workflow Execution deletes against the active Temporal Service. How that delete reaches passive Temporal +Services depends on the Temporal Server version: + +| Temporal Server version | Delete replication behavior | +| --- | --- | +| v1.30 and earlier | Explicit deletes do not replicate. | +| v1.31 | Set `history.enableDeleteWorkflowExecutionReplication` to `true` on every Temporal Service. The setting affects new deletes and does not remove Workflow Executions that were deleted earlier. | +| v1.32 and later | Explicit deletes replicate automatically. The dynamic configuration setting was removed. | + +This applies to `DeleteWorkflowExecution`, including the individual deletes performed by a batch delete. Retention +cleanup is not replicated; each Temporal Service deletes closed Workflow Executions according to the replicated +Namespace retention setting. An administrative force delete with `tdbg workflow delete` is also local to the Temporal +Service against which you run it. + +When explicit delete replication is unavailable or disabled, a Workflow Execution deleted early on the active Temporal +Service remains on each passive Temporal Service until its own retention cleanup runs. This can make a passive SQL +database temporarily larger than the active database. + +### Monitor passive database cleanup + +Each Temporal Service independently runs retention cleanup and the History scavenger. Small, stable database-size +differences are expected. Investigate a passive SQL database that continues to grow relative to the active database, +especially when the difference is concentrated in `executions`, `current_executions`, `history_tree`, or +`history_node`. + +Monitor the History scavenger with `scavenger_skips`, `scavenger_success`, and `scavenger_errors`. The scavenger skips a +History branch until it is older than `worker.historyScannerDataMinAge`, which defaults to 60 days. If leftover History +is accumulating after failed or manual deletes, lower this dynamic configuration to a duration appropriate for your +retention and recovery requirements. For example, use `1h` while draining a known backlog or a value of up to one week +to remove routine leftovers sooner. The scavenger runs every 12 hours, so the new value takes effect during its next +scheduled pass. + +Do not delete rows directly from the persistence database. Removing an execution row without its History pointer can +strand History branches. Use Temporal APIs to delete whole Workflow Executions and let the History scavenger remove +branches whose Workflow Executions no longer exist. + +### Configure custom Search Attributes + +Custom Search Attributes must be available on every Temporal Service before a failover: + +- With SQL Visibility, custom Search Attribute aliases are part of Namespace metadata and replicate to passive Temporal + Services. +- With Elasticsearch Visibility, custom Search Attributes are stored in each Temporal Service's Elasticsearch metadata + and index. Create the same Search Attributes on every Temporal Service. + +Use the same Visibility store type across the replication group. A mixed SQL and Elasticsearch configuration does not +store custom Search Attributes in the same way and is not a validated deployment pattern. + +### How to delete a Global Namespace + +Deleting a Global Namespace from the active Temporal Service does not delete the Namespace record from passive +Temporal Services. To remove it completely, disconnect the passive Temporal Services before deleting any copy: + +1. On the active Temporal Service, replace the Namespace's replication Cluster list with only the active Temporal + Service: + + ```bash + temporal operator namespace update \ + --address ActiveClusterFrontendAddress \ + --namespace YourNamespaceName \ + --cluster ActiveCluster + ``` + + Before continuing, describe the Namespace against each passive frontend and confirm that its Temporal Service is no + longer in the replication Cluster list. + +2. Delete the Namespace from the active Temporal Service: + + ```bash + temporal operator namespace delete \ + --address ActiveClusterFrontendAddress \ + --namespace YourNamespaceName + ``` + +3. Delete the disconnected copy from each former passive Temporal Service: + + ```bash + temporal operator namespace delete \ + --address PassiveClusterFrontendAddress \ + --namespace YourNamespaceName + ``` + +Confirm that `temporal operator namespace describe` returns `Namespace not found` from every frontend. Removing a +Temporal Service from the Namespace replication configuration does not clean up that Temporal Service's existing copy. + ## How to set up Multi-Cluster Replication {/* #set-up-multi-cluster-replication */} The [Multi-Cluster Replication](/self-hosted-guide/multi-cluster-replication) feature asynchronously replicates Workflow Execution Event Histories from active Clusters to other passive Clusters, and can be enabled by setting the appropriate values in the `clusterMetadata` section of your configuration file. diff --git a/docs/production-deployment/self-hosted-guide/upgrade-server.mdx b/docs/production-deployment/self-hosted-guide/upgrade-server.mdx index 31a5f95430..16adfb8926 100644 --- a/docs/production-deployment/self-hosted-guide/upgrade-server.mdx +++ b/docs/production-deployment/self-hosted-guide/upgrade-server.mdx @@ -226,3 +226,19 @@ We recommend preparing a staging Temporal Service and then doing the following t simulation load logic. 4. Upgrade the server. 5. Now do the same to the live environment Temporal Service. + +### Upgrade a Multi-Cluster Replication deployment + +Upgrade each Temporal Service separately because each one has its own database schema and server deployment. Do not +run a Namespace handover while either Temporal Service is being upgraded. + +Upgrade the passive Temporal Service before the active Temporal Service. This keeps the replication receiver at the +same or a newer version than the sender: + +1. Upgrade the passive Temporal Service's schema and server deployment. +2. Wait for `replication_tasks_lag` and `replication_latency` to return to their normal ranges. +3. Confirm that `replication_stream_stuck` and `replication_dlq_enqueue_failed` are not increasing. +4. Upgrade the active Temporal Service's schema and server deployment. + +For additional readiness checks, see [How to prepare for a Namespace +failover](/self-hosted-guide/multi-cluster-replication#how-to-prepare-for-a-namespace-failover). diff --git a/docs/references/cluster-metrics.mdx b/docs/references/cluster-metrics.mdx index 13eaacfa49..cb34e9c88f 100644 --- a/docs/references/cluster-metrics.mdx +++ b/docs/references/cluster-metrics.mdx @@ -264,6 +264,82 @@ Number of Workflows that successfully completed. Number of Workflows that timed out before completing execution. +## Multi-Cluster Replication metrics + +Use these metrics together to determine whether a passive Temporal Service is caught up and ready for a planned +Namespace handover. For the complete list of emitted metrics, see +[`metric_defs.go`](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go). + +### `replication_stream_stuck` + +Counts replication streams that stop making forward progress. A non-zero rate requires investigation before a planned +handover. + +### `replication_stream_error` + +Counts errors that close or reconnect a replication stream. Connection recycling can produce errors while replication +remains healthy, so evaluate this metric with `replication_stream_stuck`, `replication_tasks_lag`, and +`replication_latency`. + +### `replication_dlq_enqueue_failed` + +Counts failures to write an unprocessable replication task to the replication dead-letter queue. A non-zero rate can +mean a replication task was not retained for later recovery. + +### `replication_tasks_lag` + +A histogram of the difference between the source replication task identifier and the destination's acknowledged task +identifier. A rising value means the passive Temporal Service is falling behind. + +### `replication_latency` + +A histogram of the end-to-end time between replication task generation on the active Temporal Service and application +on the passive Temporal Service. + +### `replication_tasks_recv_backlog` + +A histogram of replication tasks buffered by the passive Temporal Service while waiting to be applied. A sustained +increase can trigger receiver flow control and stop the source from sending more tasks. + +### `replication_tasks_applied` + +Counts replication tasks that the destination Temporal Service applies. After a handover, use this metric on the former +active Temporal Service to verify that the reverse replication stream is processing tasks. + +### `task_errors_discarded` + +Counts standby tasks discarded after their required Event History does not arrive before the configured discard delay. +A non-zero rate can indicate that affected Workflow Executions will need task recovery after failover. + +### `catchup_ready_shard_count` + +A gauge of History shards that meet a `namespace-handover-v2` Workflow's initial catchup conditions. The metric is +tagged with the Namespace and destination Temporal Service. + +### `catchup_not_ready_shard_count` + +A gauge of History shards that do not yet meet a `namespace-handover-v2` Workflow's initial catchup conditions. This +value must reach zero before the Workflow enters the drain phase. + +### `handover_ready_shard_count` + +A gauge of History shards that have completely drained the Namespace's replication tasks during the handover phase. +The value must reach `numHistoryShards` before the handover timeout. + +### `handover_wait_latency` + +A histogram of how long API requests wait for a Namespace to leave the `handover` replication state. + +### `client_redirection_requests` + +Counts requests that a passive Temporal Service forwards to the active Temporal Service. After a handover, an increase +on the former active Temporal Service confirms that connected Clients and Workers are being forwarded. + +### `client_redirection_errors` + +Counts forwarded requests that return an error. Use the `error_type` tag to distinguish connectivity, throttling, and +unsupported-forwarding failures. + ## Nexus metrics These metrics pertain to Nexus Operations.