From f3c02c8bc4e21ff5f4de896dc4b5a72dc691766a Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 15 Sep 2026 20:13:14 -0700 Subject: [PATCH 1/3] Document Temporal Cloud write bandwidth limits --- docs/evaluate/temporal-cloud/limits.mdx | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/evaluate/temporal-cloud/limits.mdx b/docs/evaluate/temporal-cloud/limits.mdx index 304d1c9d0d..1bd8ed21be 100644 --- a/docs/evaluate/temporal-cloud/limits.mdx +++ b/docs/evaluate/temporal-cloud/limits.mdx @@ -262,6 +262,22 @@ For example, There is no limit on the number of resource IDs or actions within a single permission pair. Applying a permission to all resources of a type counts as one permission regardless of how many resources exist in your account. +### Write bandwidth limit + +Temporal Cloud limits the sustained volume of data that each Namespace can write. This limit applies to operations that write Workflow state, including Workflow starts, Signals, Updates, Activity Heartbeats, and task completions. + +The write bandwidth limit scales with the Namespace APS limit. A higher APS limit includes a proportionally higher write bandwidth limit. + +A Namespace can reach this limit while staying below its APS limit. Larger payloads use more write bandwidth per Action. When the limit is reached, write requests return `ResourceExhausted`. Temporal SDKs retry these requests according to their request retry policy. Continued throttling increases latency and can cause a client call to fail after its retries expire. + +Keep payloads small for workloads with sustained write traffic. Use [External Storage](/external-storage) for large data and pass a reference in the payload. + +#### Example + +Two workloads each perform 400 Actions per second. One writes 5 KB per Action. The other writes 500 KB per Action. The second workload uses about 100 times more write bandwidth and can reach the bandwidth limit while both workloads remain below the same APS limit. + +If the Namespace APS limit doubles from 500 to 1,000, its write bandwidth limit also doubles. + ## Nexus Endpoint level ### Nexus Endpoints limits From fead31c4f94cafb732baf8e2d5d585e3371defe4 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 15 Sep 2026 20:22:10 -0700 Subject: [PATCH 2/3] Remove bandwidth limit example --- docs/evaluate/temporal-cloud/limits.mdx | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/evaluate/temporal-cloud/limits.mdx b/docs/evaluate/temporal-cloud/limits.mdx index 1bd8ed21be..4f73389551 100644 --- a/docs/evaluate/temporal-cloud/limits.mdx +++ b/docs/evaluate/temporal-cloud/limits.mdx @@ -272,12 +272,6 @@ A Namespace can reach this limit while staying below its APS limit. Larger paylo Keep payloads small for workloads with sustained write traffic. Use [External Storage](/external-storage) for large data and pass a reference in the payload. -#### Example - -Two workloads each perform 400 Actions per second. One writes 5 KB per Action. The other writes 500 KB per Action. The second workload uses about 100 times more write bandwidth and can reach the bandwidth limit while both workloads remain below the same APS limit. - -If the Namespace APS limit doubles from 500 to 1,000, its write bandwidth limit also doubles. - ## Nexus Endpoint level ### Nexus Endpoints limits From 5971bc154138a6a0a4837c13a2ab42a9d3f6bb19 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 15 Sep 2026 20:52:00 -0700 Subject: [PATCH 3/3] Document write bandwidth metrics --- .../metrics/openmetrics/metrics-reference.mdx | 22 +++++++++++++++++++ docs/cloud/service-health.mdx | 8 +++++++ docs/evaluate/temporal-cloud/limits.mdx | 2 ++ 3 files changed, 32 insertions(+) diff --git a/docs/cloud/metrics/openmetrics/metrics-reference.mdx b/docs/cloud/metrics/openmetrics/metrics-reference.mdx index ebdbbd97cf..12605b456c 100644 --- a/docs/cloud/metrics/openmetrics/metrics-reference.mdx +++ b/docs/cloud/metrics/openmetrics/metrics-reference.mdx @@ -590,6 +590,28 @@ The current number of open workflows in a namespace. **Type**: Value +#### namespace\_bandwidth\_bytes + +The volume of data in write requests that count toward the Namespace write bandwidth limit, in bytes per second. See [Monitor write bandwidth](/cloud/service-health#monitor-write-bandwidth) for guidance on monitoring usage against the limit. + +| Label | Description | +| ----- | ----- | +| `operation` | The name of the write operation | +| `is_background` | Whether the write operation was background | + +**Type**: Rate + +#### namespace\_bandwidth\_throttled\_bytes + +The volume of data in write requests throttled by the Namespace write bandwidth limit, in bytes per second. A value greater than zero means write requests were throttled during the aggregation window. + +| Label | Description | +| ----- | ----- | +| `operation` | The name of the throttled write operation | +| `is_background` | Whether the throttled write operation was background | + +**Type**: Rate + #### temporal\_cloud\_v1\_total\_action\_count The total number of actions performed per second. Actions with `is_background=false` are counted toward the ``temporal_cloud_v1_action_limit``. diff --git a/docs/cloud/service-health.mdx b/docs/cloud/service-health.mdx index bc295a8755..fb2a2be650 100644 --- a/docs/cloud/service-health.mdx +++ b/docs/cloud/service-health.mdx @@ -175,6 +175,14 @@ for `temporal_cloud_v1_total_action_count` at a 50% threshold of the `temporal_c or directly when throttling is detected as a value greater than zero for `temporal_cloud_v1_total_action_throttled_count`. This logic can also be used to automatically scale [Temporal Resource Units](/cloud/capacity-modes#provisioned-capacity) up or down as needed. Some workloads choose to exceed limits and accept throttling because they are not latency sensitive. +### Monitor write bandwidth + +Use [`namespace_bandwidth_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_bytes) to monitor the volume of write data that counts toward the Namespace write bandwidth limit. The metric reports bytes per second averaged over each one-minute aggregation window. + +Use [`namespace_bandwidth_throttled_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_throttled_bytes) to detect write bandwidth throttling. A value greater than zero means that write requests were throttled during the aggregation window. Break down the metric by `operation` to find the write operations that contributed to the throttling. + +The write bandwidth limit scales with the Namespace APS limit. Increasing the APS limit increases the write bandwidth limit proportionally. Monitor Action usage and write bandwidth together because a Namespace with large payloads can reach its write bandwidth limit while remaining below its APS limit. + ### Provisioned capacity utilization For Namespaces in [provisioned capacity](/cloud/capacity-modes#provisioned-capacity) mode, the limit and count metrics also reveal the lower bound: how much of your reserved capacity you are actually using. diff --git a/docs/evaluate/temporal-cloud/limits.mdx b/docs/evaluate/temporal-cloud/limits.mdx index 4f73389551..dabe2768e0 100644 --- a/docs/evaluate/temporal-cloud/limits.mdx +++ b/docs/evaluate/temporal-cloud/limits.mdx @@ -268,6 +268,8 @@ Temporal Cloud limits the sustained volume of data that each Namespace can write The write bandwidth limit scales with the Namespace APS limit. A higher APS limit includes a proportionally higher write bandwidth limit. +Use the [`namespace_bandwidth_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_bytes) and [`namespace_bandwidth_throttled_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_throttled_bytes) metrics to monitor write bandwidth usage and throttling. See [Monitor write bandwidth](/cloud/service-health#monitor-write-bandwidth) for guidance. + A Namespace can reach this limit while staying below its APS limit. Larger payloads use more write bandwidth per Action. When the limit is reached, write requests return `ResourceExhausted`. Temporal SDKs retry these requests according to their request retry policy. Continued throttling increases latency and can cause a client call to fail after its retries expire. Keep payloads small for workloads with sustained write traffic. Use [External Storage](/external-storage) for large data and pass a reference in the payload.