diff --git a/docs/cloud/metrics/openmetrics/metrics-reference.mdx b/docs/cloud/metrics/openmetrics/metrics-reference.mdx index ebdbbd97cf..12605b456c 100644 --- a/docs/cloud/metrics/openmetrics/metrics-reference.mdx +++ b/docs/cloud/metrics/openmetrics/metrics-reference.mdx @@ -590,6 +590,28 @@ The current number of open workflows in a namespace. **Type**: Value +#### namespace\_bandwidth\_bytes + +The volume of data in write requests that count toward the Namespace write bandwidth limit, in bytes per second. See [Monitor write bandwidth](/cloud/service-health#monitor-write-bandwidth) for guidance on monitoring usage against the limit. + +| Label | Description | +| ----- | ----- | +| `operation` | The name of the write operation | +| `is_background` | Whether the write operation was background | + +**Type**: Rate + +#### namespace\_bandwidth\_throttled\_bytes + +The volume of data in write requests throttled by the Namespace write bandwidth limit, in bytes per second. A value greater than zero means write requests were throttled during the aggregation window. + +| Label | Description | +| ----- | ----- | +| `operation` | The name of the throttled write operation | +| `is_background` | Whether the throttled write operation was background | + +**Type**: Rate + #### temporal\_cloud\_v1\_total\_action\_count The total number of actions performed per second. Actions with `is_background=false` are counted toward the ``temporal_cloud_v1_action_limit``. diff --git a/docs/cloud/service-health.mdx b/docs/cloud/service-health.mdx index bc295a8755..fb2a2be650 100644 --- a/docs/cloud/service-health.mdx +++ b/docs/cloud/service-health.mdx @@ -175,6 +175,14 @@ for `temporal_cloud_v1_total_action_count` at a 50% threshold of the `temporal_c or directly when throttling is detected as a value greater than zero for `temporal_cloud_v1_total_action_throttled_count`. This logic can also be used to automatically scale [Temporal Resource Units](/cloud/capacity-modes#provisioned-capacity) up or down as needed. Some workloads choose to exceed limits and accept throttling because they are not latency sensitive. +### Monitor write bandwidth + +Use [`namespace_bandwidth_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_bytes) to monitor the volume of write data that counts toward the Namespace write bandwidth limit. The metric reports bytes per second averaged over each one-minute aggregation window. + +Use [`namespace_bandwidth_throttled_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_throttled_bytes) to detect write bandwidth throttling. A value greater than zero means that write requests were throttled during the aggregation window. Break down the metric by `operation` to find the write operations that contributed to the throttling. + +The write bandwidth limit scales with the Namespace APS limit. Increasing the APS limit increases the write bandwidth limit proportionally. Monitor Action usage and write bandwidth together because a Namespace with large payloads can reach its write bandwidth limit while remaining below its APS limit. + ### Provisioned capacity utilization For Namespaces in [provisioned capacity](/cloud/capacity-modes#provisioned-capacity) mode, the limit and count metrics also reveal the lower bound: how much of your reserved capacity you are actually using. diff --git a/docs/evaluate/temporal-cloud/limits.mdx b/docs/evaluate/temporal-cloud/limits.mdx index 304d1c9d0d..dabe2768e0 100644 --- a/docs/evaluate/temporal-cloud/limits.mdx +++ b/docs/evaluate/temporal-cloud/limits.mdx @@ -262,6 +262,18 @@ For example, There is no limit on the number of resource IDs or actions within a single permission pair. Applying a permission to all resources of a type counts as one permission regardless of how many resources exist in your account. +### Write bandwidth limit + +Temporal Cloud limits the sustained volume of data that each Namespace can write. This limit applies to operations that write Workflow state, including Workflow starts, Signals, Updates, Activity Heartbeats, and task completions. + +The write bandwidth limit scales with the Namespace APS limit. A higher APS limit includes a proportionally higher write bandwidth limit. + +Use the [`namespace_bandwidth_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_bytes) and [`namespace_bandwidth_throttled_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_throttled_bytes) metrics to monitor write bandwidth usage and throttling. See [Monitor write bandwidth](/cloud/service-health#monitor-write-bandwidth) for guidance. + +A Namespace can reach this limit while staying below its APS limit. Larger payloads use more write bandwidth per Action. When the limit is reached, write requests return `ResourceExhausted`. Temporal SDKs retry these requests according to their request retry policy. Continued throttling increases latency and can cause a client call to fail after its retries expire. + +Keep payloads small for workloads with sustained write traffic. Use [External Storage](/external-storage) for large data and pass a reference in the payload. + ## Nexus Endpoint level ### Nexus Endpoints limits