Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/cloud/metrics/openmetrics/metrics-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -590,6 +590,28 @@ The current number of open workflows in a namespace.

**Type**: Value

#### namespace\_bandwidth\_bytes

The volume of data in write requests that count toward the Namespace write bandwidth limit, in bytes per second. See [Monitor write bandwidth](/cloud/service-health#monitor-write-bandwidth) for guidance on monitoring usage against the limit.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the write operation |
| `is_background` | Whether the write operation was background |

**Type**: Rate

#### namespace\_bandwidth\_throttled\_bytes

The volume of data in write requests throttled by the Namespace write bandwidth limit, in bytes per second. A value greater than zero means write requests were throttled during the aggregation window.

| Label | Description |
| ----- | ----- |
| `operation` | The name of the throttled write operation |
| `is_background` | Whether the throttled write operation was background |

**Type**: Rate

#### temporal\_cloud\_v1\_total\_action\_count

The total number of actions performed per second. Actions with `is_background=false` are counted toward the ``temporal_cloud_v1_action_limit``.
Expand Down
8 changes: 8 additions & 0 deletions docs/cloud/service-health.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,14 @@ for `temporal_cloud_v1_total_action_count` at a 50% threshold of the `temporal_c
or directly when throttling is detected as a value greater than zero for `temporal_cloud_v1_total_action_throttled_count`. This logic can also be used to automatically scale [Temporal
Resource Units](/cloud/capacity-modes#provisioned-capacity) up or down as needed. Some workloads choose to exceed limits and accept throttling because they are not latency sensitive.

### Monitor write bandwidth

Use [`namespace_bandwidth_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_bytes) to monitor the volume of write data that counts toward the Namespace write bandwidth limit. The metric reports bytes per second averaged over each one-minute aggregation window.

Use [`namespace_bandwidth_throttled_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_throttled_bytes) to detect write bandwidth throttling. A value greater than zero means that write requests were throttled during the aggregation window. Break down the metric by `operation` to find the write operations that contributed to the throttling.

The write bandwidth limit scales with the Namespace APS limit. Increasing the APS limit increases the write bandwidth limit proportionally. Monitor Action usage and write bandwidth together because a Namespace with large payloads can reach its write bandwidth limit while remaining below its APS limit.

### Provisioned capacity utilization

For Namespaces in [provisioned capacity](/cloud/capacity-modes#provisioned-capacity) mode, the limit and count metrics also reveal the lower bound: how much of your reserved capacity you are actually using.
Expand Down
12 changes: 12 additions & 0 deletions docs/evaluate/temporal-cloud/limits.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,18 @@ For example,

There is no limit on the number of resource IDs or actions within a single permission pair. Applying a permission to all resources of a type counts as one permission regardless of how many resources exist in your account.

### Write bandwidth limit

Temporal Cloud limits the sustained volume of data that each Namespace can write. This limit applies to operations that write Workflow state, including Workflow starts, Signals, Updates, Activity Heartbeats, and task completions.

The write bandwidth limit scales with the Namespace APS limit. A higher APS limit includes a proportionally higher write bandwidth limit.

Use the [`namespace_bandwidth_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_bytes) and [`namespace_bandwidth_throttled_bytes`](/cloud/metrics/openmetrics/metrics-reference#namespace_bandwidth_throttled_bytes) metrics to monitor write bandwidth usage and throttling. See [Monitor write bandwidth](/cloud/service-health#monitor-write-bandwidth) for guidance.

A Namespace can reach this limit while staying below its APS limit. Larger payloads use more write bandwidth per Action. When the limit is reached, write requests return `ResourceExhausted`. Temporal SDKs retry these requests according to their request retry policy. Continued throttling increases latency and can cause a client call to fail after its retries expire.

Keep payloads small for workloads with sustained write traffic. Use [External Storage](/external-storage) for large data and pass a reference in the payload.

## Nexus Endpoint level

### Nexus Endpoints limits
Expand Down