From 1657a08af87a9a636e1bcb7b3cafbf6c057a6823 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 16:30:44 -0700 Subject: [PATCH 01/26] docs: correct fairness dispatch semantics --- docs/best-practices/multi-tenant-patterns.mdx | 12 +- .../downstream-rate-limiting.mdx | 2 +- docs/design-patterns/fairness.mdx | 295 +++--------------- docs/design-patterns/index.mdx | 2 +- docs/design-patterns/priority-task-queues.mdx | 4 +- .../qos-throughput-patterns.mdx | 6 +- docs/develop/task-queue-priority-fairness.mdx | 33 +- .../PriorityFairnessWalkthrough/HowItWorks.js | 6 +- .../PriorityFairnessWalkthrough/Overview.js | 2 +- 9 files changed, 79 insertions(+), 283 deletions(-) diff --git a/docs/best-practices/multi-tenant-patterns.mdx b/docs/best-practices/multi-tenant-patterns.mdx index 417808b4b9..c69e674605 100644 --- a/docs/best-practices/multi-tenant-patterns.mdx +++ b/docs/best-practices/multi-tenant-patterns.mdx @@ -58,14 +58,14 @@ This is the recommended pattern for most use cases. Each tenant gets dedicated T ### 2. Single Task Queue with Fairness -**Use a single [Task Queue](/task-queue) with [Fairness keys](/develop/task-queue-priority-fairness#task-queue-fairness) to distribute work across tenants.** +**Use a single [Task Queue](/task-queue) with [fairness keys](/develop/task-queue-priority-fairness#task-queue-fairness) to distribute work across tenants.** -This pattern uses [Task Queue Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) to manage multi-tenant workloads within a single Task Queue. Each tenant is assigned a fairness key, and fairness weights control how much of the Task Queue's capacity each tenant receives. +This pattern uses [Task Queue Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) to manage multi-tenant workloads within a single Task Queue. Each tenant is assigned a fairness key, and fairness weights control its share of Task dispatches when multiple tenants have backlogged Tasks. -You can also set per-fairness-key rate limits (requests per second) to cap individual tenant throughput, preventing any single tenant from consuming too much capacity. +You can also set per-fairness-key rate limits to cap an individual tenant's dispatch rate. **Pros:** -- Priority and Fairness keys and weights can be adjusted without redeployment +- Priority and fairness keys and weights can be adjusted without redeployment - Onboarding new tenants doesn't require spinning up additional [Workers](/workers) - Simpler Worker topology than per-tenant Task Queues @@ -130,8 +130,8 @@ need its own service accounts, API keys, dashboards, or rate limits. If that is | | Task Queues per tenant | Fairness-based | Shared Workflow / Separate Activity TQs | Namespace per tenant | |---|---|---|---|---| -| **Isolation** | Task Queue level | Probabilistic (weighted) | Activity-level only | Complete | -| **Noisy neighbor protection** | Strong | Weight-based throttling | Activity-level | Full — separate rate limits | +| **Isolation** | Task Queue level | Weighted dispatch | Activity-level only | Complete | +| **Noisy neighbor protection** | Strong | Dispatch-based | Activity-level | Full, with separate rate limits | | **Worker management** | Moderate — config per tenant | Simple — single Task Queue | Moderate | High — Worker pool per tenant | | **Onboarding new tenants** | Config update and restart | Set fairness and priority values (no new Workers) | Config update and Worker restart | New Namespace and Worker pool | | **Observability** | Per-Task Queue metrics | Per-Task Queue metrics | Mixed | Per-Namespace | diff --git a/docs/design-patterns/downstream-rate-limiting.mdx b/docs/design-patterns/downstream-rate-limiting.mdx index c40f2c2b09..48bf073970 100644 --- a/docs/design-patterns/downstream-rate-limiting.mdx +++ b/docs/design-patterns/downstream-rate-limiting.mdx @@ -285,7 +285,7 @@ The concurrency slots (`MaxConcurrentActivityExecutionSize`, `MaxConcurrentWorkf ### Patterns - **[Priority Task Queues](/design-patterns/priority-task-queues)**: Route work to separate queues by urgency, with different concurrency budgets per tier. -- **[Fairness](/design-patterns/fairness)**: Give each tenant an equal throughput share when multiple tenants share capacity. +- **[Fairness](/design-patterns/fairness)**: Distribute Task dispatches among tenants that share a Task Queue. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. ### Guides diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 90e3e024dd..3a0f423137 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -1,301 +1,98 @@ --- id: fairness title: "Fairness" -description: "Distributes Worker capacity evenly across tenants or users so that a burst from one caller does not starve the others." +description: "Distributes Task dispatches across tenants or users so that a burst from one caller does not starve others." --- -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - :::info[TLDR] -Assign a `FairnessKey` and weight to Workflows and Activities so each tenant or group receives the **correct proportional share of Worker capacity** on a shared Task Queue. Use this when a high-volume caller would otherwise starve other tenants without requiring separate queues per tenant. +Assign a fairness key and weight to Workflows and Activities so each tenant or group receives a proportional share of Task dispatches on a shared Task Queue. Use this when a high-volume caller would otherwise starve other tenants. ::: ## Overview -The Fairness pattern distributes Worker capacity proportionally across tenants or user groups within a single Task Queue so that a burst from one caller cannot starve others. Each group is assigned a fairness key and an optional weight; the Temporal matching service dispatches tasks in weighted round-robin order across all keys. +The Fairness pattern distributes Task dispatches across tenants or user groups within a shared Task Queue. Each group has a fairness key and an optional weight. The Temporal matching service uses weighted fair dispatch to select the next Task within a priority level. + +Fairness applies only to Task dispatch. It does not account for Task duration or resource use. ## Problem -When multiple tenants (for example, customers) share a single Task Queue, a high-volume tenant can fill the queue and occupy all Worker slots. Other tenants receive no service until the dominant tenant's backlog drains. This starvation violates throughput guarantees and makes latency for lower-volume tenants unpredictable under burst conditions. +When multiple tenants share a Task Queue, a high-volume tenant can fill the backlog and dominate dispatch. Tasks from other tenants can wait behind that backlog, which makes their latency unpredictable under load. -The classic workaround—assigning one Task Queue per tenant—scales poorly: each new tenant requires a new Worker deployment, idle capacity on low-traffic tenants cannot be used by busy ones, and queue management complexity grows with tenant count. +Using one Task Queue per tenant avoids a shared backlog, but adds Task Queue, Worker, and routing configuration for every tenant. ## Solution -Temporal's native Fairness feature lets you assign a `FairnessKey` (a string identifier such as a tenant name or tier) and an optional `FairnessWeight` (a positive float, default 1.0) to Workflows, Activities, and Child Workflows. The Temporal matching service creates a virtual queue for each key and dispatches tasks in proportion to their weights. A single shared Worker pool serves all keys; no extra queues or routing logic is required. +Assign a fairness key to each tenant or group and, when needed, a fairness weight. Tasks with the same fairness key compete for dispatch as a group. A single Worker pool can serve all keys. -For example, assigning weights of 5.0, 3.0, and 2.0 to `premium`, `basic`, and `free` tiers causes 50% of dispatched tasks to come from `premium`, 30% from `basic`, and 20% from `free`—regardless of backlog depth. Within a single fairness key, tasks are dispatched in FIFO order. +For example, assigning weights of 5.0, 3.0, and 2.0 causes approximately 50% of dispatched Tasks to come from `premium`, 30% from `basic`, and 20% from `free` when all three groups have backlogged Tasks. Within a fairness key, Tasks at the same priority are dispatched in first-in-first-out (FIFO) order. ```mermaid flowchart TD - WA["Workflow\nfairness_key=tenant-big\n(weight 1.0)"] --> TQ["my-task-queue"] - WB["Workflow\nfairness_key=tenant-mid\n(weight 1.0)"] --> TQ - WC["Workflow\nfairness_key=tenant-small\n(weight 1.0)"] --> TQ - TQ --> VQ1["Virtual Queue\ntenant-big"] - TQ --> VQ2["Virtual Queue\ntenant-mid"] - TQ --> VQ3["Virtual Queue\ntenant-small"] - VQ1 -->|round-robin| W["Shared Workers"] - VQ2 -->|round-robin| W - VQ3 -->|round-robin| W - W --> DS["Downstream\nService"] + WA["Workflow\nfairness_key=tenant-big\nweight=1.0"] --> TQ["my-task-queue"] + WB["Workflow\nfairness_key=tenant-mid\nweight=1.0"] --> TQ + WC["Workflow\nfairness_key=tenant-small\nweight=1.0"] --> TQ + TQ --> VQ1["Fairness group\ntenant-big"] + TQ --> VQ2["Fairness group\ntenant-mid"] + TQ --> VQ3["Fairness group\ntenant-small"] + VQ1 -->|weighted dispatch| W["Shared Workers"] + VQ2 -->|weighted dispatch| W + VQ3 -->|weighted dispatch| W ``` The following describes each step in the diagram: -1. Workflows start with a `FairnessKey` matching the tenant or group identity. -2. The Temporal matching service routes each task to the corresponding virtual queue inside the single Task Queue. -3. Workers poll the Task Queue and receive tasks in weighted round-robin order across all fairness keys. -4. Tenant-big's large backlog does not prevent tenant-mid or tenant-small from receiving service. +1. Workflows start with a fairness key that identifies their tenant or group. +2. Tasks with the same fairness key form a fairness group within the Task Queue. +3. When more than one group has backlogged Tasks, the matching service dispatches Tasks according to their weights. +4. A group can use all available dispatches when the other groups have no backlogged Tasks. ## Implementation -### Enable fairness - -**Temporal Cloud:** Navigate to the Namespace's Overview page in the UI and activate the Fairness toggle. Fairness is a paid feature in Temporal Cloud. - -**Self-hosted Temporal:** Set `matching.enableFairness` to `true` in the [dynamic config](/temporal-service/configuration#dynamic-configuration) for the relevant Task Queues or Namespaces. - -### Set fairness key and weight at Workflow start - - - - -```python -from temporalio.common import Priority - -handle = await client.start_workflow( - ProcessOrder.run, - id="process-order-wf", - task_queue="my-task-queue", - priority=Priority( - fairness_key="tenant-a", - fairness_weight=2.0, - ), -) -``` - - - - -```go -we, err := c.ExecuteWorkflow( - context.Background(), - client.StartWorkflowOptions{ - ID: "process-order-wf", - TaskQueue: "my-task-queue", - Priority: temporal.Priority{ - FairnessKey: "tenant-a", - FairnessWeight: 2.0, - }, - }, - ProcessOrder, -) -``` - - - - -```java -WorkflowOptions options = WorkflowOptions.newBuilder() - .setWorkflowId("process-order-wf") - .setTaskQueue("my-task-queue") - .setPriority(Priority.newBuilder() - .setFairnessKey("tenant-a") - .setFairnessWeight(2.0f) - .build()) - .build(); -ProcessOrder workflow = client.newWorkflowStub(ProcessOrder.class, options); -WorkflowClient.start(workflow::run); -``` - - - - -### Set fairness key and weight on Activities - -Activities inherit the parent Workflow's fairness key and weight. Override them in `ActivityOptions` when an Activity should belong to a different fairness group than its Workflow. Each field (`priority_key`, `fairness_key`, `fairness_weight`) is resolved independently in this order: Task Queue weight overrides (highest precedence), value set explicitly in the options, value inherited from the calling Workflow, then the default. Workflows started with Continue-As-New inherit the current execution's priority values unless you pass explicit values. See [Inheritance](/develop/task-queue-priority-fairness#inheritance) in the Temporal docs for the full resolution diagram. - - - - -```python -from temporalio.common import Priority - -# inside the workflow -result = await workflow.execute_activity( - process_for_tenant, - tenant_request, - start_to_close_timeout=timedelta(minutes=1), - priority=Priority( - fairness_key="tenant-a", - fairness_weight=2.0, - ), -) -``` - - - - -```go -ao := workflow.ActivityOptions{ - StartToCloseTimeout: time.Minute, - Priority: temporal.Priority{ - FairnessKey: "tenant-a", - FairnessWeight: 2.0, - }, -} -ctx = workflow.WithActivityOptions(ctx, ao) -err := workflow.ExecuteActivity(ctx, ProcessForTenant, req).Get(ctx, nil) -``` - - - - -```java -ActivityOptions options = ActivityOptions.newBuilder() - .setStartToCloseTimeout(Duration.ofMinutes(1)) - .setPriority(Priority.newBuilder() - .setFairnessKey("tenant-a") - .setFairnessWeight(2.0f) - .build()) - .build(); -TenantActivity activity = Workflow.newActivityStub(TenantActivity.class, options); -activity.processForTenant(request); -``` - - - - -### Set queue-level and per-key rate limits via CLI - -You can rate-limit the entire Task Queue and set a default per-fairness-key limit. The per-key limit is scaled by the fairness weight for that key, so a key with weight 2.5 and a default per-key limit of 10 gets an effective limit of 25 tasks/second. - -```sh -temporal task-queue config set \ - --task-queue my-task-queue \ - --task-queue-type activity \ - --namespace my-namespace \ - --queue-rps-limit 500 \ - --queue-rps-limit-reason "overall limit" \ - --fairness-key-rps-limit-default 33.3 \ - --fairness-key-rps-limit-reason "per-key limit" -``` - -### Override fairness weights via CLI - -When it is more convenient to manage weights through configuration than to embed them in client code, you can override weights for up to 1000 keys per Task Queue. Overrides take precedence over the weight attached to a task's options and can be updated without a code deploy. - -```sh -temporal task-queue config set \ - --task-queue my-task-queue \ - --task-queue-type workflow \ - --namespace my-namespace \ - --fairness-key-weight premium=5.0 \ - --fairness-key-weight basic=3.0 \ - --fairness-key-weight free=2.0 -``` - -### Use priority and fairness together - -Priority and Fairness can be combined. Priority determines which sub-queue (1–5) a task enters; Fairness determines the dispatch order within each priority level. Set both `PriorityKey` and `FairnessKey` on the same options object. - - - +Fairness must be enabled for the Namespace. Set fairness keys and weights on Workflows, Activities, or Child Workflows. Activities and Child Workflows inherit these values unless you override them. -```python -from temporalio.common import Priority - -handle = await client.start_workflow( - ChargeCustomer.run, - id="charge-customer-wf", - task_queue="my-task-queue", - priority=Priority( - priority_key=1, - fairness_key="tenant-a", - fairness_weight=2.0, - ), -) -``` - - - - -```go -we, err := c.ExecuteWorkflow( - context.Background(), - client.StartWorkflowOptions{ - ID: "charge-customer-wf", - TaskQueue: "my-task-queue", - Priority: temporal.Priority{ - PriorityKey: 1, - FairnessKey: "tenant-a", - FairnessWeight: 2.0, - }, - }, - ChargeCustomer, -) -``` - - - - -```java -WorkflowOptions options = WorkflowOptions.newBuilder() - .setWorkflowId("charge-customer-wf") - .setTaskQueue("my-task-queue") - .setPriority(Priority.newBuilder() - .setPriorityKey(1) - .setFairnessKey("tenant-a") - .setFairnessWeight(2.0f) - .build()) - .build(); -``` - - - +See [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) for setup, SDK examples, inheritance, Task Queue configuration, and limitations. ## When to use -This pattern is a good fit for multi-tenant applications where large tenants should not block small tenants, for workloads that need proportional capacity allocation across groups without hard rate limits, and when the set of tenants or groups is dynamic (new keys can be introduced without deploying new Workers). For a broader look at multi-tenancy strategies in Temporal, see [Multi-Tenant Patterns](/best-practices/multi-tenant-patterns). +Use this pattern when multiple tenants or workload groups share a Task Queue and need weighted dispatch under load. It works well when tenants are added often because fairness keys do not require separate Task Queues or Worker configuration. -It is not a good fit when absolute throughput isolation is required (dedicated queues per tenant remain or [task queue priorities](/design-patterns/priority-task-queues) are the appropriate choice). +Fairness does not provide exact dispatch ratios, concurrency limits, or compute isolation. Use Activity Task Queue rate limits for throughput caps. Use separate Task Queues with dedicated Worker pools and compute resources for hard isolation. Use [Priority Task Queues](/design-patterns/priority-task-queues) to order urgent work ahead of less urgent work. ## Benefits and trade-offs -A single Worker pool serves all tenants; idle capacity from a low-traffic tenant automatically benefits high-traffic tenants rather than going to waste. New tenants require no Worker deployment—add a fairness key and Temporal starts dispatching their tasks immediately. Weights can be updated via CLI without redeploying application code. +Fairness is work-conserving. A group can use all available dispatches when no other group has a backlog. New groups can start using the same Task Queue without changes to the Worker deployment. -Fairness requires explicit enablement on Temporal Cloud and self-hosted deployments. Accuracy can degrade with a very large number of fairness keys. Fairness weight applies at schedule time, not dispatch time: changing a weight does not retroactively reorder tasks already in the backlog. +Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, Worker Versioning, and short time windows. Tasks with different runtimes can consume different amounts of Worker capacity even when their dispatch shares match their weights. ## Comparison with alternatives -| Approach | Tenant isolation | Dynamic tenants | Shares idle capacity | Complexity | -| :--- | :--- | :--- | :--- | :--- | -| Temporal FairnessKey (native) | Soft | Yes | Yes | Low | -| Dedicated queue per tenant | Hard | No | No | Medium | -| Single shared queue (no control) | None | Yes | Yes | Lowest | -| External queue with per-tenant consumer groups | Hard | Yes | No | High | +| Approach | Backlog dispatch | Worker capacity isolation | Tenant onboarding | +| :--- | :--- | :--- | :--- | +| Fairness on a shared Task Queue | Weighted across backlogged groups | None | Assign a fairness key | +| Task Queue per tenant with shared compute | Separate tenant backlogs | None | Add Task Queue and Worker configuration | +| Task Queue per tenant with dedicated compute | Separate tenant backlogs | Yes | Deploy and configure dedicated Workers | +| Shared Task Queue without Fairness | No tenant-aware ordering | None | No additional configuration | ## Best practices -- **Use stable, consistent naming for fairness keys.** Use account IDs or tenant slugs rather than display names. Key names cannot be changed retroactively on tasks already in the backlog. -- **Combine Priority and Fairness for multi-class, multi-tenant workloads.** Priority separates urgent from batch work; Fairness prevents any single tenant from dominating within each priority level. -- **Monitor queue depth by fairness key.** Sustained backlog growth for a particular key means its weight fraction of Worker capacity cannot drain its submission rate. +- **Use stable fairness keys.** Use account identifiers or tenant slugs instead of display names. +- **Use one weight for each key.** Conflicting weights for the same key have unspecified behavior. +- **Combine Priority and Fairness for mixed workloads.** Use Priority for urgency classes and Fairness for tenants within each class. +- **Measure results by tenant.** Track submitted, started, and completed Tasks and latency in application telemetry. ## Common pitfalls -- **Expecting Fairness to reorder the existing backlog.** Fairness weight is evaluated at schedule time. Enabling Fairness on a Namespace with an existing backlog drains that backlog in its original order first; the fairness-aware dispatch mode takes effect only for newly submitted tasks. -- **Using Fairness as a hard rate limiter.** Fairness controls proportional dispatch but does not cap the absolute throughput of any one key. For hard throughput caps, combine Fairness with per-fairness-key RPS limits via the CLI. -- **Unkeyed tasks bypassing Fairness.** Tasks without a `FairnessKey` are grouped under an implicit empty-string key and participate in round-robin dispatch alongside named keys with a weight of 1.0. They do not bypass Fairness and compete as one group. -- **Task Queue partitioning reducing accuracy.** Task Queues are internally partitioned and tasks are distributed to partitions randomly, which can interfere with fair dispatch proportions. If your workload requires higher accuracy, contact Temporal Support to configure a single-partition Task Queue. -- **Assuming Fairness applies across Worker Versioning boundaries.** When using Worker Versioning and moving Workflows between versions, Priority still applies across versions but Fairness is only guaranteed within tasks originally queued on the same Worker version. Tasks moved from one version to another may not dispatch in fairness order relative to tasks on the destination version. -- **Expecting consistent fairness immediately after a server restart.** Fairness ordering is preserved across restarts for the most active keys. Less active keys may briefly dispatch new tasks ahead of their existing backlog until ordering normalizes. -- **Expecting the running task mix to immediately reflect fair dispatch.** Fairness governs which task is dispatched next; it does not account for tasks already running on Workers. The mix of in-flight tasks at any moment may not match the configured weight ratios. +- **Expecting exact ratios.** Weights control dispatch proportions over time when multiple groups have backlogged Tasks. Results vary across partitions and short time windows. +- **Treating Fairness as Worker capacity control.** Fairness does not account for running Tasks or their resource use. A group with longer Tasks can consume more Worker time than its dispatch share suggests. +- **Using Fairness as a hard rate limiter.** Fairness does not cap absolute throughput. Use whole-queue or per-fairness-key RPS limits on Activity Task Queues for dispatch-rate caps. +- **Expecting every Task to pass through fair dispatch.** A Task can dispatch immediately when it synchronously matches an idle poller. Eagerly dispatched Tasks bypass matching. + +See [Limitations of Fairness](/develop/task-queue-priority-fairness#limitations-of-fairness) for partitioning, Worker Versioning, backlog migration, and Task Queue reload behavior. ## Related ### Patterns -- **[Priority Task Queues](/design-patterns/priority-task-queues)**: Order tasks by urgency level within the same Task Queue using `PriorityKey`. -- **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap absolute throughput to a downstream service with a queue RPS setting. +- **[Priority Task Queues](/design-patterns/priority-task-queues)**: Order Tasks by urgency within the same Task Queue using `PriorityKey`. +- **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap dispatch throughput to a downstream service with a Task Queue RPS setting. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. diff --git a/docs/design-patterns/index.mdx b/docs/design-patterns/index.mdx index 93d19f808f..c4f1b8ea57 100644 --- a/docs/design-patterns/index.mdx +++ b/docs/design-patterns/index.mdx @@ -221,7 +221,7 @@ Having these patterns in your toolbox helps you solve recurring problems in a ba href: "/design-patterns/fairness", icon: "fairness-icon.svg", title: "Fairness", - description: "Distributes Worker capacity evenly across tenants or users so that a burst from one caller does not starve the others.", + description: "Distributes Task dispatches across tenants or users so that a burst from one caller does not starve the others.", }, ]} /> diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index 322f76df00..5fd040a128 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -201,7 +201,7 @@ temporal workflow start \ This pattern is a good fit when your system mixes time-sensitive operations (payment processing, user-facing requests) with background or batch work (reporting, data imports, inventory management), and you want urgent tasks to proceed even during periods of high load. It also works well when you need to mark urgent tasks that should override normal processing—for example, triggering immediate re-runs of failed critical tasks. -It is not a good fit when all work is effectively equal in urgency, when a continuously replenished high-priority backlog could starve lower-priority work indefinitely, or when you need hard capacity isolation between tiers (see dedicated queues per tier as a supplementary measure). If your concern is prioritizing work amongst tenants or customers, consider the [Fairness](/design-patterns/fairness) pattern, which distributes capacity proportionally using weighted fairness keys rather than strict ordering. +It is not a good fit when all work is effectively equal in urgency, when a continuously replenished high-priority backlog could starve lower-priority work indefinitely, or when you need hard capacity isolation between tiers (see dedicated queues per tier as a supplementary measure). For tenant-aware dispatch, use the [Fairness](/design-patterns/fairness) pattern. ## Benefits and trade-offs @@ -238,7 +238,7 @@ Lower-priority tasks are blocked until all higher-priority tasks have started. I ### Patterns -- **[Fairness](/design-patterns/fairness)**: Distribute capacity proportionally across tenants within a priority level using fairness keys. +- **[Fairness](/design-patterns/fairness)**: Distribute dispatches proportionally across tenants within a priority level using fairness keys. - **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap absolute throughput to a downstream service regardless of task priority. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. diff --git a/docs/design-patterns/qos-throughput-patterns.mdx b/docs/design-patterns/qos-throughput-patterns.mdx index 0e355c1383..6d9173db01 100644 --- a/docs/design-patterns/qos-throughput-patterns.mdx +++ b/docs/design-patterns/qos-throughput-patterns.mdx @@ -1,12 +1,12 @@ --- id: qos-throughput-patterns title: "QoS & Throughput Patterns" -description: "Pattern selection guide for controlling execution rate, protecting downstream services from overload, and ensuring fair capacity distribution across tenants." +description: "Pattern selection guide for controlling execution rate, protecting downstream services from overload, and distributing dispatch across tenants." --- import PatternCards from '@site/src/components/PatternCards'; -These patterns control how fast work executes, protect downstream services from overload, and make sure no single caller or tenant monopolizes Worker capacity at the expense of others. +These patterns control how fast work executes, protect downstream services from overload, and keep one caller or tenant from dominating dispatch. ## Patterns in this section @@ -27,7 +27,7 @@ These patterns control how fast work executes, protect downstream services from href: "/design-patterns/fairness", icon: "fairness-icon.svg", title: "Fairness", - description: "Distributes Worker capacity evenly across tenants or users so a burst from one caller does not starve the others.", + description: "Distributes Task dispatches across tenants or users so a burst from one caller does not starve the others.", }, ]} /> diff --git a/docs/develop/task-queue-priority-fairness.mdx b/docs/develop/task-queue-priority-fairness.mdx index 5f06f9422b..30ddc8daf0 100644 --- a/docs/develop/task-queue-priority-fairness.mdx +++ b/docs/develop/task-queue-priority-fairness.mdx @@ -210,17 +210,16 @@ await Workflow.ExecuteChildWorkflowAsync( Task Queue Fairness lets you distribute Tasks based on _fairness keys_ and _fairness weights_ within a Task Queue. -Each fairness key creates its own "virtual queue", allowing you to organize Tasks into logical groups like tenants, applications, or workload types. These virtual queues operate using a round-robin dispatch mechanism, meaning the system cycles through each fairness key in turn when selecting the next Task to dispatch. This prevents any single fairness key from hogging Worker capacity, even if one key has a much larger backlog than the others. +Fairness keys organize Tasks into groups such as tenants, applications, or workload types. Weighted fair dispatch selects among groups that have backlogged Tasks. This keeps a group that fills the backlog from dominating dispatch. -By default, each fairness key is weighted equally in the round-robin, with a _fairness weight_ of 1.0. This behavior can be customized by assigning a different fairness weight to a key. For example, Tasks belonging to a fairness key with a weight of 2.0 will be dispatched twice as often as keys with the default weight. +Each fairness key has a default _fairness weight_ of 1.0. You can assign a different weight to a key. When two groups have backlogged Tasks, a group with a weight of 2.0 receives approximately twice as many dispatches as a group with a weight of 1.0. ### When to use Fairness Fairness is intended to address common situations like: - Multi-tenant applications with big and small tenants where small tenants shouldn't be blocked by big ones. -- Assigning Tasks to different capacity bands and then, for example, dispatching 80% from one band and 20% from another - without limiting overall capacity when one band is empty. +- Assigning Tasks to weighted groups and dispatching approximately 80% from one group and 20% from another when both have backlogged Tasks. It sequences Tasks in the Task Queue probabilistically using a weighted distribution based on: @@ -231,7 +230,7 @@ It sequences Tasks in the Task Queue probabilistically using a weighted distribu As an example, imagine a workload with three tenants, _tenant-big_, _tenant-mid_, _tenant-small_, that have varying numbers of Tasks at all times. Your _tenant-big_ has a large number of Tasks that can overwhelm your Task Queue and prevent _tenant-mid_ and _tenant-small_ from running their Tasks. With Fairness, you can give each tenant a different -fairness key to make sure _tenant-big_ doesn't use all of the Task Queue resources and block the others. In this case, +fairness key to make sure _tenant-big_ doesn't dominate dispatch and block the others. In this case, _tenant-mid_ and _tenant-small_ will have Tasks run in between _tenant-big_ Tasks so that they are executed "fairly". ### How to use Fairness @@ -242,11 +241,11 @@ To enable Fairness for a Namespace in Temporal Cloud, navigate to the Namespace' If you're self-hosting Temporal, set `matching.enableFairness` to `true` in the [dynamic config](/temporal-service/configuration#dynamic-configuration) on the relevant Task Queues or Namespaces. -To use Fairness, you need to set fairness keys and optionally fairness weights at the Workflow, Activity, or Child Workflow level. Tasks with different fairness keys are dispatched in proportion to their fairness weights. For example, if you weight _premium-tier_ at 5.0, _basic-tier_ at 3.0, and _free-tier_ at 2.0, then 50% of dispatched Tasks come from _premium-tier_, 30% from _basic-tier_, and 20% from _free-tier_. If there are Tasks in the Task Queue backlog that have the same fairness key, then they're dispatched in [FIFO order](/task-queue#task-ordering). +To use Fairness, set fairness keys and optionally fairness weights at the Workflow, Activity, or Child Workflow level. Tasks with different fairness keys are dispatched in proportion to their fairness weights. For example, weights of 5.0 for _premium-tier_, 3.0 for _basic-tier_, and 2.0 for _free-tier_ cause approximately 50% of dispatched Tasks to come from _premium-tier_, 30% from _basic-tier_, and 20% from _free-tier_ when all three groups have backlogged Tasks. Within the same priority level and Task Queue partition, Tasks in the backlog with the same fairness key are dispatched in [FIFO order](/task-queue#task-ordering). You can set a Workflow's fairness key and weight via the CLI like so: @@ -653,7 +652,7 @@ var handle = await Client.StartWorkflowAsync( -Tasks that do not have a `fairness_key` set are grouped together under an implicit empty-string key. All unkeyed Tasks share this single default bucket and participate in the same round-robin dispatch alongside named fairness keys, with a default weight of 1.0. This means Fairness adoption can be incremental: you can assign fairness keys to some tenants but not others. Unkeyed Tasks do not bypass Fairness; they compete as one group alongside all explicitly keyed Tasks. +Tasks that do not have a `fairness_key` set are grouped under an implicit empty-string key with a default weight of 1.0. The group participates in weighted fair dispatch alongside named fairness keys. This lets you adopt Fairness incrementally. :::info @@ -664,20 +663,20 @@ There should only be one fairness weight assigned to each fairness key within a ### Choosing between Priority, Fairness, and both - **Priority alone** when you need strict priority ordering - for example, separating real-time Tasks from batch Tasks. -- **Fairness alone** when you need tier or tenant isolation so no group is starved, but you don't need to preempt any group ahead of another. +- **Fairness alone** when you need weighted dispatch among tiers or tenants without ordering one group ahead of another. - **Both** when you have a tiered SLA hierarchy - Priority for the broad tier (for example, paid vs. free), Fairness for per-tenant equity within a tier. When you use Priority and Fairness together, the next Task to dispatch is chosen by walking three rules in order: 1. **Priority tier (strict).** Tasks at a higher priority always dispatch before tasks at lower priorities, regardless of fairness keys or weights. -2. **Fairness key within a tier (weighted).** Within a priority tier, each fairness key is a virtual queue. Keys are dispatched proportional to their weights - a key with weight 2.0 is dispatched twice as often as one with weight 1.0. +2. **Fairness key within a tier (weighted).** Within a priority tier, Tasks are dispatched according to the weights of their fairness keys. When both groups are backlogged, a key with weight 2.0 receives approximately twice as many dispatches as a key with weight 1.0. 3. **FIFO within a key.** Tasks that share a priority tier _and_ fairness key dispatch in the order they were enqueued. These rules apply within a Task Queue partition. ### Inheritance @@ -730,12 +729,11 @@ temporal task-queue config set \ **Whole queue rate limits:** applies to the whole queue regardless of the fairness key. This is the same setting as is exposed through the [Worker Options](/develop/worker-tuning-reference#io-configuration-options) in the SDKs, and when set via the API, takes precedence over the limit set through Worker Options. -**Fairness key rate limits:** The per-fairness-key rate limit works in conjunction with Task Queue Fairness. If you think of Fairness as dividing the queue into one virtual queue for each key, then the per-fairness-key rate limit is a limit on each individual virtual queue. Some important notes on the per-fairness-key limit: +**Fairness key rate limits:** The per-fairness-key rate limit caps the dispatch rate for each fairness key. Some important notes on the per-fairness-key limit: - The whole queue limit and per-fairness-key limit may be set independently: none, one or the other, or both may be set. If both are set, then the more restrictive one applies. -- The per-fairness-key limit for a key is scaled by the fairness weight assigned to that key. So if the per-fairness-key limit for a queue is set to 10, then all keys with the default weight (1.0) will have a limit of 10 tasks/second. But if a particular key is given a weight of 2.5, then the per-key rate limit for that key will be 25 tasks/second. -- Since the dispatch rate for each key should be proportional to its weight, if any key is hitting the per-key limit, then nearly all of them are. The way it works is if the next Task to be dispatched hits the per-key limit, then dispatch will wait until it can go. -- Usually there isn't actually any blocking, but there can be when the fairness weight for a key is changed between when a Task is scheduled and when it's dispatched. If the fairness weight for a key is lowered, for example, the new lower per-key rate limit will be respected. Since those Tasks were originally scheduled with the higher rate, they will block other Tasks as they're dispatched. This limitation will be improved in the future. +- The per-fairness-key limit for a key is scaled by the fairness weight assigned to that key. If the default limit is 10 Tasks per second, a key with weight 1.0 has a limit of 10 Tasks per second and a key with weight 2.5 has a limit of 25 Tasks per second. +- When a Task would exceed its key's rate limit, matching can skip it and dispatch another eligible Task. ### Fairness weight overrides @@ -760,7 +758,8 @@ To unset a single key's override, pass `key=default`. To clear all overrides on - There isn't a limit on the number of fairness keys you can use, but their accuracy can degrade as you add more. - Fairness is enforced within a single Task Queue [partition](/task-queue#task-ordering). When a Task Queue's partitions are imbalanced, Fairness may not appear to hold, since it applies only within individual partitions. Depending on your use case, you can reach out to Temporal Support to get your Task Queues set to a single partition. -- The fairness weight applies at schedule time, not at dispatch time. So it only affects newly-scheduled Tasks, not currently backlogged ones. This means if you need to throttle a single fairness key in the existing backlog of Tasks, you won't be able to. +- A Task's fairness weight is recorded when the Task is scheduled. Changing a weight in application code affects newly scheduled Tasks, not the current backlog. - When you use Worker Versioning and you're moving Workflows from one version to another, Priority will still apply between versions. Fairness isn't guaranteed between versions. For example, you may have Tasks that were originally queued on Worker version _alpha_, Tasks that were queued on Worker version _beta_, and some Tasks were moved from _alpha_ to _beta_. Fairness is only guaranteed when Tasks are originally queued on the same Worker version. So there might be some discrepancies on the Tasks moved from _alpha_ to _beta_. -- During server restarts, Temporal preserves fairness state for the top 100 keys. Other keys rebuild their fairness state as new Tasks arrive, which can temporarily distort weighted dispatch. Fairness pass dithering reduces this distortion by spreading keys' initial positions according to their weights. It can reduce FIFO-like ordering among equal-weight keys. To enable fairness pass dithering, [contact Temporal Support](/cloud/support#support-ticket). +- When a Task Queue partition reloads or changes ownership, Temporal restores fairness state for up to 100 keys by default. Other keys rebuild their state as new Tasks arrive, which can temporarily distort weighted dispatch. Fairness pass dithering spreads the initial positions of these keys according to their weights. It can reduce FIFO-like ordering among equal-weight keys. To enable fairness pass dithering, [contact Temporal Support](/cloud/support#support-ticket). - Fairness doesn't consider Task executions that have already been dispatched to Workers. As a result, fair dispatch may not be immediately visible in the mix of Tasks currently running on Workers. +- Tasks that synchronously match an available poller can bypass backlog ordering. [Eagerly dispatched Tasks](/develop/worker-performance#eager-task-execution) also bypass matching. diff --git a/src/components/elements/PriorityFairnessWalkthrough/HowItWorks.js b/src/components/elements/PriorityFairnessWalkthrough/HowItWorks.js index 48af60e667..837e6019e4 100644 --- a/src/components/elements/PriorityFairnessWalkthrough/HowItWorks.js +++ b/src/components/elements/PriorityFairnessWalkthrough/HowItWorks.js @@ -8,8 +8,8 @@ const STEPS = [ body: <>Tasks at a higher priority (lower number) are always dispatched before tasks at a lower priority. Every task at priority 1 is dispatched before any task at priority 2, and so on., }, { - title: 'Fairness distributes capacity within the tier', - body: <>Within a priority tier, tasks are dispatched proportionally by fairnessWeight using a weighted round-robin mechanism. This prevents any single fairnessKey from hogging Worker capacity, even if it has a deep backlog., + title: 'Fairness distributes dispatches within the tier', + body: <>Within a priority tier, weighted fair dispatch selects among fairness keys with backlogged Tasks. This prevents one fairness key from filling the backlog and dominating dispatch., }, { title: <>No fairnessKey means FIFO within the tier, @@ -32,7 +32,7 @@ export default function HowItWorks({ onNext }) {

When a Worker polls for the next task, Temporal applies two rules in sequence: Priority - determines which tier goes first, and Fairness distributes capacity among keys within each + determines which tier goes first, and Fairness distributes dispatches among keys within each tier.

diff --git a/src/components/elements/PriorityFairnessWalkthrough/Overview.js b/src/components/elements/PriorityFairnessWalkthrough/Overview.js index dfc4528f6e..8424ac4869 100644 --- a/src/components/elements/PriorityFairnessWalkthrough/Overview.js +++ b/src/components/elements/PriorityFairnessWalkthrough/Overview.js @@ -62,7 +62,7 @@ export default function Overview({ onNext }) {

Without Fairness, tasks at the same priority dispatch in FIFO order, so a backlog-heavy - tenant can hog Worker capacity and delay everyone else at that level. Fairness groups + tenant can fill the backlog and dominate dispatch at that level. Fairness groups tasks by a fairness key and dispatches them proportionally by fairness weight. A key with weight 2.0 is dispatched twice as often as a key with the default weight of 1.0.

From b8adf9428095f59b37123986ce80255645c96272 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 17:29:35 -0700 Subject: [PATCH 02/26] docs: preserve Fairness term capitalization --- docs/best-practices/multi-tenant-patterns.mdx | 4 ++-- docs/design-patterns/fairness.mdx | 20 +++++++++---------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/best-practices/multi-tenant-patterns.mdx b/docs/best-practices/multi-tenant-patterns.mdx index c69e674605..8d1949876c 100644 --- a/docs/best-practices/multi-tenant-patterns.mdx +++ b/docs/best-practices/multi-tenant-patterns.mdx @@ -58,14 +58,14 @@ This is the recommended pattern for most use cases. Each tenant gets dedicated T ### 2. Single Task Queue with Fairness -**Use a single [Task Queue](/task-queue) with [fairness keys](/develop/task-queue-priority-fairness#task-queue-fairness) to distribute work across tenants.** +**Use a single [Task Queue](/task-queue) with [Fairness keys](/develop/task-queue-priority-fairness#task-queue-fairness) to distribute work across tenants.** This pattern uses [Task Queue Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) to manage multi-tenant workloads within a single Task Queue. Each tenant is assigned a fairness key, and fairness weights control its share of Task dispatches when multiple tenants have backlogged Tasks. You can also set per-fairness-key rate limits to cap an individual tenant's dispatch rate. **Pros:** -- Priority and fairness keys and weights can be adjusted without redeployment +- Priority and Fairness keys and weights can be adjusted without redeployment - Onboarding new tenants doesn't require spinning up additional [Workers](/workers) - Simpler Worker topology than per-tenant Task Queues diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 3a0f423137..5b3c774ebb 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -5,12 +5,12 @@ description: "Distributes Task dispatches across tenants or users so that a burs --- :::info[TLDR] -Assign a fairness key and weight to Workflows and Activities so each tenant or group receives a proportional share of Task dispatches on a shared Task Queue. Use this when a high-volume caller would otherwise starve other tenants. +Assign a Fairness key and weight to Workflows and Activities so each tenant or group receives a proportional share of Task dispatches on a shared Task Queue. Use this when a high-volume caller would otherwise starve other tenants. ::: ## Overview -The Fairness pattern distributes Task dispatches across tenants or user groups within a shared Task Queue. Each group has a fairness key and an optional weight. The Temporal matching service uses weighted fair dispatch to select the next Task within a priority level. +The Fairness pattern distributes Task dispatches across tenants or user groups within a shared Task Queue. Each group has a Fairness key and an optional weight. The Temporal matching service uses weighted fair dispatch to select the next Task within a priority level. Fairness applies only to Task dispatch. It does not account for Task duration or resource use. @@ -22,9 +22,9 @@ Using one Task Queue per tenant avoids a shared backlog, but adds Task Queue, Wo ## Solution -Assign a fairness key to each tenant or group and, when needed, a fairness weight. Tasks with the same fairness key compete for dispatch as a group. A single Worker pool can serve all keys. +Assign a Fairness key to each tenant or group and, when needed, a Fairness weight. Tasks with the same Fairness key compete for dispatch as a group. A single Worker pool can serve all keys. -For example, assigning weights of 5.0, 3.0, and 2.0 causes approximately 50% of dispatched Tasks to come from `premium`, 30% from `basic`, and 20% from `free` when all three groups have backlogged Tasks. Within a fairness key, Tasks at the same priority are dispatched in first-in-first-out (FIFO) order. +For example, assigning weights of 5.0, 3.0, and 2.0 causes approximately 50% of dispatched Tasks to come from `premium`, 30% from `basic`, and 20% from `free` when all three groups have backlogged Tasks. Within a Fairness key, Tasks at the same priority are dispatched in first-in-first-out (FIFO) order. ```mermaid flowchart TD @@ -41,20 +41,20 @@ flowchart TD The following describes each step in the diagram: -1. Workflows start with a fairness key that identifies their tenant or group. -2. Tasks with the same fairness key form a fairness group within the Task Queue. +1. Workflows start with a Fairness key that identifies their tenant or group. +2. Tasks with the same Fairness key form a fairness group within the Task Queue. 3. When more than one group has backlogged Tasks, the matching service dispatches Tasks according to their weights. 4. A group can use all available dispatches when the other groups have no backlogged Tasks. ## Implementation -Fairness must be enabled for the Namespace. Set fairness keys and weights on Workflows, Activities, or Child Workflows. Activities and Child Workflows inherit these values unless you override them. +Fairness must be enabled for the Namespace. Set Fairness keys and weights on Workflows, Activities, or Child Workflows. Activities and Child Workflows inherit these values unless you override them. See [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) for setup, SDK examples, inheritance, Task Queue configuration, and limitations. ## When to use -Use this pattern when multiple tenants or workload groups share a Task Queue and need weighted dispatch under load. It works well when tenants are added often because fairness keys do not require separate Task Queues or Worker configuration. +Use this pattern when multiple tenants or workload groups share a Task Queue and need weighted dispatch under load. It works well when tenants are added often because Fairness keys do not require separate Task Queues or Worker configuration. Fairness does not provide exact dispatch ratios, concurrency limits, or compute isolation. Use Activity Task Queue rate limits for throughput caps. Use separate Task Queues with dedicated Worker pools and compute resources for hard isolation. Use [Priority Task Queues](/design-patterns/priority-task-queues) to order urgent work ahead of less urgent work. @@ -68,14 +68,14 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, | Approach | Backlog dispatch | Worker capacity isolation | Tenant onboarding | | :--- | :--- | :--- | :--- | -| Fairness on a shared Task Queue | Weighted across backlogged groups | None | Assign a fairness key | +| Fairness on a shared Task Queue | Weighted across backlogged groups | None | Assign a Fairness key | | Task Queue per tenant with shared compute | Separate tenant backlogs | None | Add Task Queue and Worker configuration | | Task Queue per tenant with dedicated compute | Separate tenant backlogs | Yes | Deploy and configure dedicated Workers | | Shared Task Queue without Fairness | No tenant-aware ordering | None | No additional configuration | ## Best practices -- **Use stable fairness keys.** Use account identifiers or tenant slugs instead of display names. +- **Use stable Fairness keys.** Use account identifiers or tenant slugs instead of display names. - **Use one weight for each key.** Conflicting weights for the same key have unspecified behavior. - **Combine Priority and Fairness for mixed workloads.** Use Priority for urgency classes and Fairness for tenants within each class. - **Measure results by tenant.** Track submitted, started, and completed Tasks and latency in application telemetry. From f5ae7e68568096c0c11d471cfd7b3aede77fb3bd Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 17:33:30 -0700 Subject: [PATCH 03/26] docs: simplify Priority guidance --- docs/design-patterns/fairness.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 5b3c774ebb..e8ae207065 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -56,7 +56,7 @@ See [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness#tas Use this pattern when multiple tenants or workload groups share a Task Queue and need weighted dispatch under load. It works well when tenants are added often because Fairness keys do not require separate Task Queues or Worker configuration. -Fairness does not provide exact dispatch ratios, concurrency limits, or compute isolation. Use Activity Task Queue rate limits for throughput caps. Use separate Task Queues with dedicated Worker pools and compute resources for hard isolation. Use [Priority Task Queues](/design-patterns/priority-task-queues) to order urgent work ahead of less urgent work. +Fairness does not provide exact dispatch ratios, concurrency limits, or compute isolation. Use Activity Task Queue rate limits for throughput caps. Use separate Task Queues with dedicated Worker pools and compute resources for hard isolation. Use Priority to order urgent work ahead of less urgent work. ## Benefits and trade-offs From 76f8e443da2946794f118d897ed897503f1c81b1 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 17:35:00 -0700 Subject: [PATCH 04/26] docs: trim Fairness best practices --- docs/design-patterns/fairness.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index e8ae207065..a00b102a91 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -76,7 +76,6 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, ## Best practices - **Use stable Fairness keys.** Use account identifiers or tenant slugs instead of display names. -- **Use one weight for each key.** Conflicting weights for the same key have unspecified behavior. - **Combine Priority and Fairness for mixed workloads.** Use Priority for urgency classes and Fairness for tenants within each class. - **Measure results by tenant.** Track submitted, started, and completed Tasks and latency in application telemetry. From 566ffcdb92c0911d3e511a4f9f5ee059ee88a007 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 17:35:26 -0700 Subject: [PATCH 05/26] docs: remove unsupported Fairness monitoring guidance --- docs/design-patterns/fairness.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index a00b102a91..964ca8534e 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -77,7 +77,6 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, - **Use stable Fairness keys.** Use account identifiers or tenant slugs instead of display names. - **Combine Priority and Fairness for mixed workloads.** Use Priority for urgency classes and Fairness for tenants within each class. -- **Measure results by tenant.** Track submitted, started, and completed Tasks and latency in application telemetry. ## Common pitfalls From 1e9e636698f4f948addadac7f607fa685d6662f9 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 17:35:56 -0700 Subject: [PATCH 06/26] docs: clarify long-running Fairness tasks --- docs/design-patterns/fairness.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 964ca8534e..4962699c3d 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -81,7 +81,7 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, ## Common pitfalls - **Expecting exact ratios.** Weights control dispatch proportions over time when multiple groups have backlogged Tasks. Results vary across partitions and short time windows. -- **Treating Fairness as Worker capacity control.** Fairness does not account for running Tasks or their resource use. A group with longer Tasks can consume more Worker time than its dispatch share suggests. +- **Treating Fairness as Worker capacity control.** Fairness does not account for running Tasks or their resource use. A group with longer-running Tasks can consume more Worker time than its dispatch share suggests. - **Using Fairness as a hard rate limiter.** Fairness does not cap absolute throughput. Use whole-queue or per-fairness-key RPS limits on Activity Task Queues for dispatch-rate caps. - **Expecting every Task to pass through fair dispatch.** A Task can dispatch immediately when it synchronously matches an idle poller. Eagerly dispatched Tasks bypass matching. From bdeef114cbd863413133e050281bf67a09b202fb Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 17:43:16 -0700 Subject: [PATCH 07/26] Clarify fairness limitations --- docs/design-patterns/fairness.mdx | 15 +++++++++------ docs/develop/task-queue-priority-fairness.mdx | 2 +- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 4962699c3d..1350b6b308 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -80,12 +80,15 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, ## Common pitfalls -- **Expecting exact ratios.** Weights control dispatch proportions over time when multiple groups have backlogged Tasks. Results vary across partitions and short time windows. -- **Treating Fairness as Worker capacity control.** Fairness does not account for running Tasks or their resource use. A group with longer-running Tasks can consume more Worker time than its dispatch share suggests. -- **Using Fairness as a hard rate limiter.** Fairness does not cap absolute throughput. Use whole-queue or per-fairness-key RPS limits on Activity Task Queues for dispatch-rate caps. -- **Expecting every Task to pass through fair dispatch.** A Task can dispatch immediately when it synchronously matches an idle poller. Eagerly dispatched Tasks bypass matching. - -See [Limitations of Fairness](/develop/task-queue-priority-fairness#limitations-of-fairness) for partitioning, Worker Versioning, backlog migration, and Task Queue reload behavior. +- **Expecting exact ratios.** Weights shape dispatch proportions over time when multiple groups have backlogs. Results vary across partitions and short time windows. +- **Expecting Fairness to reorder the backlog.** Temporal records each Task's Fairness weight when the Task is scheduled. Enabling Fairness or changing weights affects only newly scheduled Tasks. +- **Expecting Tasks without a Fairness key to bypass Fairness.** These Tasks share an implicit empty-string key with a weight of 1.0. +- **Expecting Fairness across Task Queue partitions.** Each partition calculates Fairness independently. Imbalanced partitions can change overall dispatch proportions. +- **Expecting Fairness across Worker Deployment Versions.** Each version has a separate backlog. Fairness applies within each version's backlog. +- **Expecting Fairness state to be fully restored after a server restart.** By default, Temporal restores state for up to 100 Fairness keys. Other keys rebuild state as Tasks arrive. +- **Treating Fairness as Worker capacity control.** Fairness considers only dispatch. A group with longer-running Tasks can consume more Worker time than its dispatch share suggests. +- **Using Fairness as a hard rate limiter.** Fairness does not cap throughput. Use [rate limits](/develop/task-queue-priority-fairness#set-rate-limits-at-the-task-queue-level) for dispatch caps. +- **Expecting every Task to pass through fair dispatch.** Synchronous matching can send a Task directly to an idle poller. [Eager Task Execution](/develop/worker-performance#eager-task-execution) bypasses matching. ## Related diff --git a/docs/develop/task-queue-priority-fairness.mdx b/docs/develop/task-queue-priority-fairness.mdx index 30ddc8daf0..bcb73ba896 100644 --- a/docs/develop/task-queue-priority-fairness.mdx +++ b/docs/develop/task-queue-priority-fairness.mdx @@ -759,7 +759,7 @@ To unset a single key's override, pass `key=default`. To clear all overrides on - There isn't a limit on the number of fairness keys you can use, but their accuracy can degrade as you add more. - Fairness is enforced within a single Task Queue [partition](/task-queue#task-ordering). When a Task Queue's partitions are imbalanced, Fairness may not appear to hold, since it applies only within individual partitions. Depending on your use case, you can reach out to Temporal Support to get your Task Queues set to a single partition. - A Task's fairness weight is recorded when the Task is scheduled. Changing a weight in application code affects newly scheduled Tasks, not the current backlog. -- When you use Worker Versioning and you're moving Workflows from one version to another, Priority will still apply between versions. Fairness isn't guaranteed between versions. For example, you may have Tasks that were originally queued on Worker version _alpha_, Tasks that were queued on Worker version _beta_, and some Tasks were moved from _alpha_ to _beta_. Fairness is only guaranteed when Tasks are originally queued on the same Worker version. So there might be some discrepancies on the Tasks moved from _alpha_ to _beta_. +- With Worker Versioning, each version has a separate backlog. Fairness applies within each version's backlog. - When a Task Queue partition reloads or changes ownership, Temporal restores fairness state for up to 100 keys by default. Other keys rebuild their state as new Tasks arrive, which can temporarily distort weighted dispatch. Fairness pass dithering spreads the initial positions of these keys according to their weights. It can reduce FIFO-like ordering among equal-weight keys. To enable fairness pass dithering, [contact Temporal Support](/cloud/support#support-ticket). - Fairness doesn't consider Task executions that have already been dispatched to Workers. As a result, fair dispatch may not be immediately visible in the mix of Tasks currently running on Workers. - Tasks that synchronously match an available poller can bypass backlog ordering. [Eagerly dispatched Tasks](/develop/worker-performance#eager-task-execution) also bypass matching. From 1851d8ab680da7c9f50cc5cd463e7ca5f44187a7 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 17:49:29 -0700 Subject: [PATCH 08/26] Correct Priority design pattern --- .../downstream-rate-limiting.mdx | 4 +- docs/design-patterns/fairness.mdx | 2 +- docs/design-patterns/index.mdx | 4 +- docs/design-patterns/priority-task-queues.mdx | 240 +++--------------- .../qos-throughput-patterns.mdx | 6 +- 5 files changed, 48 insertions(+), 208 deletions(-) diff --git a/docs/design-patterns/downstream-rate-limiting.mdx b/docs/design-patterns/downstream-rate-limiting.mdx index 48bf073970..31f7cd7819 100644 --- a/docs/design-patterns/downstream-rate-limiting.mdx +++ b/docs/design-patterns/downstream-rate-limiting.mdx @@ -240,7 +240,7 @@ public class MyWorkflowImpl implements MyWorkflow { This pattern is a good fit when your Workflow calls a downstream service with explicit requests-per-second limits, when you need throughput enforcement that holds across many concurrent Workflow instances without per-Activity logic, and when only a subset of Activity types require throttling and others should run without restriction. -It is not a good fit when you need concurrency limits rather than throughput limits (see [Priority Task Queues](/design-patterns/priority-task-queues)), when the downstream system has no rate limit and throughput is bounded only by Workflow logic, or when all Activities require the same limit and a single shared queue suffices. +It is not a good fit when you need concurrency limits rather than throughput limits, when the downstream system has no rate limit and throughput is bounded only by Workflow logic, or when all Activities require the same limit and a single shared queue suffices. ## Benefits and trade-offs @@ -284,7 +284,7 @@ The concurrency slots (`MaxConcurrentActivityExecutionSize`, `MaxConcurrentWorkf ### Patterns -- **[Priority Task Queues](/design-patterns/priority-task-queues)**: Route work to separate queues by urgency, with different concurrency budgets per tier. +- **[Priority](/design-patterns/priority-task-queues)**: Order Task dispatches by urgency within a shared Task Queue. - **[Fairness](/design-patterns/fairness)**: Distribute Task dispatches among tenants that share a Task Queue. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 1350b6b308..4d69c12b5e 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -94,6 +94,6 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, ### Patterns -- **[Priority Task Queues](/design-patterns/priority-task-queues)**: Order Tasks by urgency within the same Task Queue using `PriorityKey`. +- **[Priority](/design-patterns/priority-task-queues)**: Order Task dispatches by urgency within the same Task Queue using a Priority key. - **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap dispatch throughput to a downstream service with a Task Queue RPS setting. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. diff --git a/docs/design-patterns/index.mdx b/docs/design-patterns/index.mdx index c4f1b8ea57..fdd7fe5958 100644 --- a/docs/design-patterns/index.mdx +++ b/docs/design-patterns/index.mdx @@ -214,8 +214,8 @@ Having these patterns in your toolbox helps you solve recurring problems in a ba { href: "/design-patterns/priority-task-queues", icon: "priority-task-queues-icon.svg", - title: "Priority Task Queues", - description: "Assigns a priority level to Workflows and Activities so that time-sensitive work executes ahead of lower-priority work within a single Task Queue.", + title: "Priority", + description: "Orders Task dispatches so urgent work moves ahead of lower-priority work on the same Task Queue.", }, { href: "/design-patterns/fairness", diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index 5fd040a128..f16084ee91 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -1,249 +1,89 @@ --- id: priority-task-queues -title: "Priority Task Queues" -description: "Assigns a priority level to Workflows and Activities so that time-sensitive work executes ahead of lower-priority work within a single Task Queue." +title: "Priority" +description: "Orders Task dispatches so urgent work moves ahead of lower-priority work on the same Task Queue." --- -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - :::info[TLDR] -Assign a `PriorityKey` (1–5) to Workflows and Activities so **high priority work executes ahead of lower priority work** on a shared Task Queue. Use this when a flood of batch or background tasks would otherwise delay high-urgency requests. +Assign a Priority key from 1 to 5 to Workflows, Activities, and Child Workflows. When Tasks are backlogged, Temporal dispatches higher-priority Tasks first. Use Priority when urgent work should move ahead of a backlog. ::: ## Overview -The Priority Task Queues pattern assigns a `PriorityKey` to Workflows, Activities, and Child Workflows so that time-sensitive work executes ahead of lower-priority work within a single Task Queue—without requiring separate queues or routing logic. +Priority orders Task dispatches within a shared Task Queue. Lower Priority key values represent higher priority, so `1` is the highest priority and `5` is the lowest. + +Priority applies to dispatch. It does not preempt running Tasks or reserve Worker capacity. ## Problem -In a shared Task Queue, tasks execute in generally first-in-first-out (FIFO) order. When a large batch of low-priority work—nightly reports, bulk imports, background processing—floods the queue before time-sensitive requests arrive, the higher-priority requests wait behind the entire batch. A single queue with no ordering mechanism gives equal treatment to all tasks regardless of business urgency. +Without Priority, backlogged Tasks are generally dispatched in first-in-first-out (FIFO) order within a Task Queue partition. A large batch can fill the backlog before urgent work arrives, delaying time-sensitive Tasks. + +Separate Task Queues can isolate the backlogs, but require more routing and Worker configuration. ## Solution -Temporal's native Priority feature lets you assign a `PriorityKey` (an integer from 1 to 5, where 1 is the highest priority and 5 is the lowest) to any Workflow, Activity, or Child Workflow. The Temporal matching service maintains a sub-queue for each priority level and exhausts all tasks at a given level before dispatching to the next. Tasks default to priority 3 when no key is set. Activities and Child Workflows inherit the parent Workflow's priority unless they set their own. +Assign a Priority key to each Workflow, Activity, or Child Workflow. Within a Task Queue partition, the Matching Service dispatches the highest-priority backlogged Tasks first. Without Fairness, Tasks with the same Priority key are dispatched in FIFO order. + +Tasks use Priority key `3` by default. Activities and Child Workflows inherit the parent Workflow's Priority key unless they set their own. ```mermaid flowchart TD - WF1["Workflow\nPriorityKey=1\n(payment)"] --> TQ["my-task-queue"] - WF2["Workflow\nPriorityKey=3\n(default)"] --> TQ - WF3["Workflow\nPriorityKey=5\n(batch report)"] --> TQ + WF1["Workflow\nPriority 1\n(payment)"] --> TQ["my-task-queue"] + WF2["Workflow\nPriority 3\n(default)"] --> TQ + WF3["Workflow\nPriority 5\n(batch report)"] --> TQ TQ --> P1["Priority 1\nsub-queue"] TQ --> P3["Priority 3\nsub-queue"] TQ --> P5["Priority 5\nsub-queue"] - P1 -->|dispatched first| W["Shared Workers"] - P3 -->|dispatched second| W - P5 -->|dispatched last| W - W --> DS["Downstream\nService"] + P1 -->|dispatch first| W["Shared Workers"] + P3 -->|dispatch next| W + P5 -->|dispatch last| W ``` -The following describes each step in the diagram: - -1. Workflows start with a `PriorityKey` in their start options. Payment workflows use priority 1; routine workflows default to 3; nightly batch reports use priority 5. -2. The Temporal matching service routes each task to the corresponding priority sub-queue inside the single Task Queue. -3. Workers poll the Task Queue and receive tasks in priority order: all priority-1 tasks are dispatched before any priority-2 task, and so on. -4. Activities and Child Workflows inherit the parent Workflow's `PriorityKey` unless they explicitly set their own. +The diagram assumes all three levels have backlogged Tasks. Lower-priority Tasks wait for higher-priority backlogs to drain. ## Implementation -Priority is enabled by default in Temporal Cloud and self-hosted Temporal. - -### Set Workflow priority at start - - - - -```python -from temporalio.common import Priority - -handle = await client.start_workflow( - ChargeCustomer.run, - id="charge-customer-wf", - task_queue="my-task-queue", - priority=Priority(priority_key=1), -) -``` - - - - -```go -we, err := c.ExecuteWorkflow( - context.Background(), - client.StartWorkflowOptions{ - ID: "charge-customer-wf", - TaskQueue: "my-task-queue", - Priority: temporal.Priority{PriorityKey: 1}, - }, - ChargeCustomer, -) -``` - - - - -```java -WorkflowOptions options = WorkflowOptions.newBuilder() - .setWorkflowId("charge-customer-wf") - .setTaskQueue("my-task-queue") - .setPriority(Priority.newBuilder().setPriorityKey(1).build()) - .build(); -ChargeCustomer workflow = client.newWorkflowStub(ChargeCustomer.class, options); -WorkflowClient.start(workflow::run); -``` - - - - -### Set Activity priority - -Activities inherit the parent Workflow's priority. Override the `PriorityKey` in `ActivityOptions` when an individual Activity should run at a different level than its Workflow. - - - - -```python -from temporalio.common import Priority - -# inside the workflow -result = await workflow.execute_activity( - process_payment, - start_to_close_timeout=timedelta(minutes=1), - priority=Priority(priority_key=1), -) -``` - - - +Priority is enabled by default in Temporal Cloud and self-hosted Temporal. Set a Priority key in Workflow start options or in Activity and Child Workflow options. -```go -ao := workflow.ActivityOptions{ - StartToCloseTimeout: time.Minute, - Priority: temporal.Priority{PriorityKey: 1}, -} -ctx = workflow.WithActivityOptions(ctx, ao) -err := workflow.ExecuteActivity(ctx, ProcessPayment).Get(ctx, nil) -``` - - - - -```java -ActivityOptions options = ActivityOptions.newBuilder() - .setStartToCloseTimeout(Duration.ofMinutes(1)) - .setPriority(Priority.newBuilder().setPriorityKey(1).build()) - .build(); -PaymentActivities activities = Workflow.newActivityStub(PaymentActivities.class, options); -activities.processPayment(); -``` - - - - -### Set Child Workflow priority - - - - -```python -from temporalio.common import Priority - -# inside the parent workflow -result = await workflow.execute_child_workflow( - ProcessOrder.run, - id="process-order-child", - task_queue="my-task-queue", - priority=Priority(priority_key=2), -) -``` - - - - -```go -cwo := workflow.ChildWorkflowOptions{ - WorkflowID: "process-order-child", - TaskQueue: "my-task-queue", - Priority: temporal.Priority{PriorityKey: 2}, -} -ctx = workflow.WithChildOptions(ctx, cwo) -err := workflow.ExecuteChildWorkflow(ctx, ProcessOrder).Get(ctx, nil) -``` - - - - -```java -ChildWorkflowOptions options = ChildWorkflowOptions.newBuilder() - .setWorkflowId("process-order-child") - .setTaskQueue("my-task-queue") - .setPriority(Priority.newBuilder().setPriorityKey(2).build()) - .build(); -ProcessOrder child = Workflow.newChildWorkflowStub(ProcessOrder.class, options); -child.run(); -``` - - - - -### Set priority via CLI - -```sh -temporal workflow start \ - --type ChargeCustomer \ - --task-queue my-task-queue \ - --workflow-id charge-customer-wf \ - --input '{"customerId":"12345"}' \ - --priority-key 1 -``` +See [Task Queue Priority](/develop/task-queue-priority-fairness#task-queue-priority) for SDK and command-line examples, inheritance behavior, and self-hosted configuration. ## When to use -This pattern is a good fit when your system mixes time-sensitive operations (payment processing, user-facing requests) with background or batch work (reporting, data imports, inventory management), and you want urgent tasks to proceed even during periods of high load. It also works well when you need to mark urgent tasks that should override normal processing—for example, triggering immediate re-runs of failed critical tasks. +Use Priority when urgent and routine work share a Task Queue and Worker pool. For example, payment Tasks can dispatch ahead of inventory updates or batch reports. -It is not a good fit when all work is effectively equal in urgency, when a continuously replenished high-priority backlog could starve lower-priority work indefinitely, or when you need hard capacity isolation between tiers (see dedicated queues per tier as a supplementary measure). For tenant-aware dispatch, use the [Fairness](/design-patterns/fairness) pattern. +Priority is a poor fit when high-priority work remains continuously backlogged because lower-priority Tasks can starve. Use separate Task Queues and Worker pools for capacity isolation. Use [Fairness](/design-patterns/fairness) for tenant-aware dispatch within a Priority level. ## Benefits and trade-offs -Native priority requires no extra queues, routing logic, or additional Worker pools. A single pool of Workers serves all priority levels, so idle capacity at low-priority levels is automatically used by higher-priority work without any additional configuration. - -Lower-priority tasks are blocked until all higher-priority tasks have started. In an environment with a continuously replenished high-priority backlog, low-priority tasks may be significantly delayed. The built-in `PriorityKey` range is 1–5; if more than five distinct levels are needed, the feature cannot accommodate them. +Priority uses one Task Queue and Worker pool without additional routing. It supports five priority levels. Strict ordering can starve lower-priority Tasks and does not provide Worker capacity isolation. ## Comparison with alternatives -| Approach | Isolation | Dynamic priority | Complexity | Scales to many priorities | -| :--- | :--- | :--- | :--- | :--- | -| Temporal PriorityKey (native) | Soft | Yes | Low | Yes (1–5 levels) | -| [Fairness](/design-patterns/fairness) | Soft | Yes | Low | Yes (unlimited keys) | -| Separate Task Queues per tier | Hard | No | Medium | No (static tiers) | -| Single queue (no control) | None | N/A | Lowest | N/A | -| External queue (Kafka, SQS) | Hard | Yes | High | Yes | +| Approach | Backlog dispatch | Worker capacity isolation | +| :--- | :--- | :--- | +| Priority on a shared Task Queue | Higher-priority Tasks first | None | +| [Fairness](/design-patterns/fairness) on a shared Task Queue | Weighted across groups within a Priority level | None | +| Separate Task Queues with shared compute | Independent backlogs | None | +| Separate Task Queues with dedicated compute | Independent backlogs | Yes | ## Best practices -- **Use no more than five priority levels.** The `PriorityKey` range is 1–5. Keep levels coarse—for example, 1 = urgent, 3 = normal, 5 = batch—rather than mapping fine-grained business importance to many values. -- **Reserve priority 1 for genuinely urgent work.** If high priority is the fallback when no priority is specified, the highest level fills with routine work and the feature provides no benefit. The default is 3 when no key is set. -- **Set `PriorityKey` at Workflow start, not inside Workflow code.** Workflow code cannot change its own priority after it starts. Set the priority in the start options before execution begins. -- **Override Activity priority deliberately.** Activities inherit the parent Workflow's priority by default. Override only when a specific Activity must run at a different level than its Workflow. -- **Monitor queue depth per priority level.** Sustained backlog growth at a priority level signals that Worker capacity is insufficient for the submitted load at that level. +- **Define a small set of levels.** For example, use `1` for urgent work, `3` for normal work, and `5` for batch work. +- **Use inheritance.** Set an Activity or Child Workflow's Priority key only when it should differ from its parent Workflow. ## Common pitfalls -- **Assigning priority 1 to all work by default.** When every caller sets the highest priority, the feature provides no ordering benefit. Establish an explicit policy for which work types qualify for each level. -- **Neglecting low-priority starvation.** Under sustained high load, priority-5 tasks may wait indefinitely. Use `ScheduleToStartTimeout` on low-priority activities to surface starvation as a visible failure. -- **Changing priority after scheduling.** `PriorityKey` is evaluated when a task enters the queue and cannot be changed while it waits. To re-prioritize an already-queued task, cancel it and reschedule with the new priority. -- **Assuming hard isolation between priority levels.** Priority controls dispatch order, not Worker capacity allocation. A priority-5 task may still consume a Worker slot that is then unavailable for a priority-1 task arriving a moment later. +- **Assigning Priority key `1` to all work.** Priority cannot order Tasks when they all use the same key. +- **Expecting Priority across Task Queue partitions.** Each partition orders its backlog independently. +- **Expecting Priority across Worker Deployment Versions.** Each version has a separate backlog. Priority applies within each version's backlog. +- **Expecting FIFO order within a Priority level when using Fairness.** Fairness controls dispatch within each Priority level. +- **Expecting every Task to pass through priority dispatch.** Synchronous matching can send a Task directly to an idle poller. [Eager Task Execution](/develop/worker-performance#eager-task-execution) bypasses matching. +- **Ignoring lower-priority starvation.** A sustained higher-priority backlog can prevent lower-priority Tasks from dispatching. ## Related ### Patterns -- **[Fairness](/design-patterns/fairness)**: Distribute dispatches proportionally across tenants within a priority level using fairness keys. -- **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap absolute throughput to a downstream service regardless of task priority. +- **[Fairness](/design-patterns/fairness)**: Distribute dispatches across tenants within a Priority level. +- **[Downstream Rate Limiting](/design-patterns/downstream-rate-limiting)**: Cap dispatch throughput to a downstream service. - **[Worker-Specific Task Queues](/design-patterns/worker-specific-taskqueue)**: Route Activities to a specific Worker host for resource or data affinity. - -### Sample code - -The official Temporal documentation provides SDK code examples for setting priority keys on Workflows, Activities, and Child Workflows across all supported languages: - -- [Task Queue Priority and Fairness — Temporal docs](/develop/task-queue-priority-fairness#task-queue-priority) diff --git a/docs/design-patterns/qos-throughput-patterns.mdx b/docs/design-patterns/qos-throughput-patterns.mdx index 6d9173db01..e06f3597b2 100644 --- a/docs/design-patterns/qos-throughput-patterns.mdx +++ b/docs/design-patterns/qos-throughput-patterns.mdx @@ -20,8 +20,8 @@ These patterns control how fast work executes, protect downstream services from { href: "/design-patterns/priority-task-queues", icon: "priority-task-queues-icon.svg", - title: "Priority Task Queues", - description: "Assigns a priority level to Workflows and Activities so time-sensitive work runs ahead of lower-priority work on the same Task Queue.", + title: "Priority", + description: "Orders Task dispatches so urgent work moves ahead of lower-priority work on the same Task Queue.", }, { href: "/design-patterns/fairness", @@ -35,7 +35,7 @@ These patterns control how fast work executes, protect downstream services from **A downstream dependency has a fixed rate limit**: use [Downstream Rate Limiting](/design-patterns/downstream-rate-limiting) to cap throughput at the Worker. -**Urgent work must not wait behind bulk work**: use [Priority Task Queues](/design-patterns/priority-task-queues). +**Urgent work should move ahead of bulk work**: use [Priority](/design-patterns/priority-task-queues). **Multiple tenants share the same Workers**: use [Fairness](/design-patterns/fairness) to keep one tenant's burst from starving others. From f3f286fe2e880ec1ac7ec1dd703a419b1b03defa Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:05:15 -0700 Subject: [PATCH 09/26] Restore multi-tenant patterns page --- docs/best-practices/multi-tenant-patterns.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/best-practices/multi-tenant-patterns.mdx b/docs/best-practices/multi-tenant-patterns.mdx index 8d1949876c..417808b4b9 100644 --- a/docs/best-practices/multi-tenant-patterns.mdx +++ b/docs/best-practices/multi-tenant-patterns.mdx @@ -60,9 +60,9 @@ This is the recommended pattern for most use cases. Each tenant gets dedicated T **Use a single [Task Queue](/task-queue) with [Fairness keys](/develop/task-queue-priority-fairness#task-queue-fairness) to distribute work across tenants.** -This pattern uses [Task Queue Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) to manage multi-tenant workloads within a single Task Queue. Each tenant is assigned a fairness key, and fairness weights control its share of Task dispatches when multiple tenants have backlogged Tasks. +This pattern uses [Task Queue Fairness](/develop/task-queue-priority-fairness#task-queue-fairness) to manage multi-tenant workloads within a single Task Queue. Each tenant is assigned a fairness key, and fairness weights control how much of the Task Queue's capacity each tenant receives. -You can also set per-fairness-key rate limits to cap an individual tenant's dispatch rate. +You can also set per-fairness-key rate limits (requests per second) to cap individual tenant throughput, preventing any single tenant from consuming too much capacity. **Pros:** - Priority and Fairness keys and weights can be adjusted without redeployment @@ -130,8 +130,8 @@ need its own service accounts, API keys, dashboards, or rate limits. If that is | | Task Queues per tenant | Fairness-based | Shared Workflow / Separate Activity TQs | Namespace per tenant | |---|---|---|---|---| -| **Isolation** | Task Queue level | Weighted dispatch | Activity-level only | Complete | -| **Noisy neighbor protection** | Strong | Dispatch-based | Activity-level | Full, with separate rate limits | +| **Isolation** | Task Queue level | Probabilistic (weighted) | Activity-level only | Complete | +| **Noisy neighbor protection** | Strong | Weight-based throttling | Activity-level | Full — separate rate limits | | **Worker management** | Moderate — config per tenant | Simple — single Task Queue | Moderate | High — Worker pool per tenant | | **Onboarding new tenants** | Config update and restart | Set fairness and priority values (no new Workers) | Config update and Worker restart | New Namespace and Worker pool | | **Observability** | Per-Task Queue metrics | Per-Task Queue metrics | Mixed | Per-Namespace | From c7cbb7beb6242e3035aecb2b9b87eef40e0c28fd Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:08:12 -0700 Subject: [PATCH 10/26] Label fairness keys in diagram --- docs/design-patterns/fairness.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 4d69c12b5e..c1a2332790 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -31,9 +31,9 @@ flowchart TD WA["Workflow\nfairness_key=tenant-big\nweight=1.0"] --> TQ["my-task-queue"] WB["Workflow\nfairness_key=tenant-mid\nweight=1.0"] --> TQ WC["Workflow\nfairness_key=tenant-small\nweight=1.0"] --> TQ - TQ --> VQ1["Fairness group\ntenant-big"] - TQ --> VQ2["Fairness group\ntenant-mid"] - TQ --> VQ3["Fairness group\ntenant-small"] + TQ --> VQ1["Fairness key\ntenant-big"] + TQ --> VQ2["Fairness key\ntenant-mid"] + TQ --> VQ3["Fairness key\ntenant-small"] VQ1 -->|weighted dispatch| W["Shared Workers"] VQ2 -->|weighted dispatch| W VQ3 -->|weighted dispatch| W From c0f9fedda1b9031f793b7a8d1eeaf31f69f03b5e Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:10:22 -0700 Subject: [PATCH 11/26] Clarify fairness scheduling model --- docs/design-patterns/fairness.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index c1a2332790..c9c90b7350 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -22,7 +22,7 @@ Using one Task Queue per tenant avoids a shared backlog, but adds Task Queue, Wo ## Solution -Assign a Fairness key to each tenant or group and, when needed, a Fairness weight. Tasks with the same Fairness key compete for dispatch as a group. A single Worker pool can serve all keys. +Assign a Fairness key to each tenant or group and, when needed, a Fairness weight. A useful mental model is one virtual queue per Fairness key, with weighted round-robin dispatch across the queues. Temporal approximates this model with stride scheduling and a count-min sketch for larger key sets. A single Worker pool can serve all keys. For example, assigning weights of 5.0, 3.0, and 2.0 causes approximately 50% of dispatched Tasks to come from `premium`, 30% from `basic`, and 20% from `free` when all three groups have backlogged Tasks. Within a Fairness key, Tasks at the same priority are dispatched in first-in-first-out (FIFO) order. @@ -42,9 +42,9 @@ flowchart TD The following describes each step in the diagram: 1. Workflows start with a Fairness key that identifies their tenant or group. -2. Tasks with the same Fairness key form a fairness group within the Task Queue. -3. When more than one group has backlogged Tasks, the matching service dispatches Tasks according to their weights. -4. A group can use all available dispatches when the other groups have no backlogged Tasks. +2. Tasks with the same Fairness key enter the same virtual queue. +3. When multiple virtual queues have backlogged Tasks, the Matching Service uses weighted round robin to choose between them. +4. One virtual queue can use all available dispatches when the others have no backlogged Tasks. ## Implementation From 08fdf6348b1edb929904d97665d3222f4e2c50f3 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:15:55 -0700 Subject: [PATCH 12/26] Describe idle capacity sharing --- docs/design-patterns/fairness.mdx | 10 +++++----- docs/design-patterns/priority-task-queues.mdx | 10 +++++----- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index c9c90b7350..a1ec557d95 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -66,12 +66,12 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, ## Comparison with alternatives -| Approach | Backlog dispatch | Worker capacity isolation | Tenant onboarding | +| Approach | Backlog dispatch | Shares idle capacity | Tenant onboarding | | :--- | :--- | :--- | :--- | -| Fairness on a shared Task Queue | Weighted across backlogged groups | None | Assign a Fairness key | -| Task Queue per tenant with shared compute | Separate tenant backlogs | None | Add Task Queue and Worker configuration | -| Task Queue per tenant with dedicated compute | Separate tenant backlogs | Yes | Deploy and configure dedicated Workers | -| Shared Task Queue without Fairness | No tenant-aware ordering | None | No additional configuration | +| Fairness on a shared Task Queue | Weighted across backlogged groups | Yes | Assign a Fairness key | +| Task Queue per tenant with shared compute | Separate tenant backlogs | Yes | Add Task Queue and Worker configuration | +| Task Queue per tenant with dedicated compute | Separate tenant backlogs | No | Deploy and configure dedicated Workers | +| Shared Task Queue without Fairness | No tenant-aware ordering | Yes | No additional configuration | ## Best practices diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index f16084ee91..e1a3f02a27 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -59,12 +59,12 @@ Priority uses one Task Queue and Worker pool without additional routing. It supp ## Comparison with alternatives -| Approach | Backlog dispatch | Worker capacity isolation | +| Approach | Backlog dispatch | Shares idle capacity | | :--- | :--- | :--- | -| Priority on a shared Task Queue | Higher-priority Tasks first | None | -| [Fairness](/design-patterns/fairness) on a shared Task Queue | Weighted across groups within a Priority level | None | -| Separate Task Queues with shared compute | Independent backlogs | None | -| Separate Task Queues with dedicated compute | Independent backlogs | Yes | +| Priority on a shared Task Queue | Higher-priority Tasks first | Yes | +| [Fairness](/design-patterns/fairness) on a shared Task Queue | Weighted across groups within a Priority level | Yes | +| Separate Task Queues with shared compute | Independent backlogs | Yes | +| Separate Task Queues with dedicated compute | Independent backlogs | No | ## Best practices From 9dabfeecd96c6f78d8227ebea7ef21f2ce53d0e1 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:17:08 -0700 Subject: [PATCH 13/26] Clarify Fairness key rate limits --- docs/design-patterns/fairness.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index a1ec557d95..74917358d6 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -87,7 +87,7 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, - **Expecting Fairness across Worker Deployment Versions.** Each version has a separate backlog. Fairness applies within each version's backlog. - **Expecting Fairness state to be fully restored after a server restart.** By default, Temporal restores state for up to 100 Fairness keys. Other keys rebuild state as Tasks arrive. - **Treating Fairness as Worker capacity control.** Fairness considers only dispatch. A group with longer-running Tasks can consume more Worker time than its dispatch share suggests. -- **Using Fairness as a hard rate limiter.** Fairness does not cap throughput. Use [rate limits](/develop/task-queue-priority-fairness#set-rate-limits-at-the-task-queue-level) for dispatch caps. +- **Using Fairness as a hard rate limiter.** Fairness by itself doesn't cap throughput. Use [Fairness key rate limits](/develop/task-queue-priority-fairness#set-rate-limits-at-the-task-queue-level) for per-key limits. - **Expecting every Task to pass through fair dispatch.** Synchronous matching can send a Task directly to an idle poller. [Eager Task Execution](/develop/worker-performance#eager-task-execution) bypasses matching. ## Related From 39e8d935a2a099dd652e77bd2cefa7b5913db4d4 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:22:06 -0700 Subject: [PATCH 14/26] Restore Priority and Fairness guidance --- docs/design-patterns/fairness.mdx | 4 +++- docs/design-patterns/priority-task-queues.mdx | 10 +++++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 74917358d6..ca5641eb75 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -54,7 +54,9 @@ See [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness#tas ## When to use -Use this pattern when multiple tenants or workload groups share a Task Queue and need weighted dispatch under load. It works well when tenants are added often because Fairness keys do not require separate Task Queues or Worker configuration. +This pattern is a good fit for multi-tenant applications where large tenants should not block small tenants. It also fits workloads that need proportional Task dispatch across groups without hard per-group limits, and applications where tenants or groups change often. New Fairness keys can be introduced without deploying new Workers. + +For a broader look at multi-tenancy strategies in Temporal, see [Multi-Tenant Patterns](/best-practices/multi-tenant-patterns). Fairness does not provide exact dispatch ratios, concurrency limits, or compute isolation. Use Activity Task Queue rate limits for throughput caps. Use separate Task Queues with dedicated Worker pools and compute resources for hard isolation. Use Priority to order urgent work ahead of less urgent work. diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index e1a3f02a27..f99f7e4318 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -49,13 +49,17 @@ See [Task Queue Priority](/develop/task-queue-priority-fairness#task-queue-prior ## When to use -Use Priority when urgent and routine work share a Task Queue and Worker pool. For example, payment Tasks can dispatch ahead of inventory updates or batch reports. +Use Priority when a Task Queue handles work with different levels of urgency. Common examples include payments or user-facing requests sharing Workers with reports, data imports, or inventory updates. When Tasks back up, Priority dispatches urgent Tasks ahead of routine Tasks. -Priority is a poor fit when high-priority work remains continuously backlogged because lower-priority Tasks can starve. Use separate Task Queues and Worker pools for capacity isolation. Use [Fairness](/design-patterns/fairness) for tenant-aware dispatch within a Priority level. +Priority also works well for exceptional Tasks that should dispatch ahead of normal work, such as an operator-triggered recovery Task. + +Priority adds little when all work has the same urgency. A sustained high-priority backlog can starve lower-priority Tasks. Use separate Task Queues with dedicated Workers and compute for capacity isolation. Use [Fairness](/design-patterns/fairness) when tenants within a Priority level need weighted shares of dispatches. ## Benefits and trade-offs -Priority uses one Task Queue and Worker pool without additional routing. It supports five priority levels. Strict ordering can starve lower-priority Tasks and does not provide Worker capacity isolation. +Priority keeps work on one Task Queue and Worker pool. Urgency levels require no separate routing or Workers. All levels share idle Worker capacity. When one level has no backlog, Tasks at other levels can use the available dispatches. + +Priority applies only to Tasks waiting for dispatch. It does not preempt Tasks that are already running. A sustained higher-priority backlog can delay lower-priority Tasks indefinitely. Priority supports five levels, from `1` for the highest priority to `5` for the lowest. ## Comparison with alternatives From 8e726974b90fea96299788a19d4dfd24ca79b9a5 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:27:19 -0700 Subject: [PATCH 15/26] Preserve Priority and Fairness narrative --- docs/design-patterns/fairness.mdx | 22 +++++++--------- docs/design-patterns/priority-task-queues.mdx | 26 ++++++++++++------- 2 files changed, 27 insertions(+), 21 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index ca5641eb75..3cc017b952 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -5,24 +5,24 @@ description: "Distributes Task dispatches across tenants or users so that a burs --- :::info[TLDR] -Assign a Fairness key and weight to Workflows and Activities so each tenant or group receives a proportional share of Task dispatches on a shared Task Queue. Use this when a high-volume caller would otherwise starve other tenants. +Assign a Fairness key and weight to Workflows and Activities so each tenant or group receives a proportional share of Task dispatches on a shared Task Queue. Use this when a high-volume caller would otherwise starve other tenants, without requiring a separate Task Queue per tenant. ::: ## Overview -The Fairness pattern distributes Task dispatches across tenants or user groups within a shared Task Queue. Each group has a Fairness key and an optional weight. The Temporal matching service uses weighted fair dispatch to select the next Task within a priority level. +The Fairness pattern distributes Task dispatches proportionally across tenants or user groups within a shared Task Queue so that a burst from one caller cannot starve others. Each group has a Fairness key and an optional weight. The Matching Service uses weighted fair dispatch to select the next Task within a Priority level. Fairness applies only to Task dispatch. It does not account for Task duration or resource use. ## Problem -When multiple tenants share a Task Queue, a high-volume tenant can fill the backlog and dominate dispatch. Tasks from other tenants can wait behind that backlog, which makes their latency unpredictable under load. +When multiple tenants, such as customers, share a Task Queue, a high-volume tenant can fill the backlog and dominate dispatch. Tasks from other tenants can wait behind that backlog, making their latency unpredictable during bursts. -Using one Task Queue per tenant avoids a shared backlog, but adds Task Queue, Worker, and routing configuration for every tenant. +The classic workaround is one Task Queue per tenant. This adds Task Queue, Worker, and routing configuration for every tenant. It can also strand idle capacity when Workers are dedicated to individual tenants. ## Solution -Assign a Fairness key to each tenant or group and, when needed, a Fairness weight. A useful mental model is one virtual queue per Fairness key, with weighted round-robin dispatch across the queues. Temporal approximates this model with stride scheduling and a count-min sketch for larger key sets. A single Worker pool can serve all keys. +Temporal's Fairness feature lets you assign a Fairness key, such as a tenant name or tier, and an optional Fairness weight to Workflows, Activities, and Child Workflows. A useful mental model is one virtual queue per Fairness key, with weighted round-robin dispatch across the queues. Temporal approximates this model with stride scheduling and a count-min sketch for larger key sets. A single shared Worker pool serves all keys, with no extra queues or routing logic. For example, assigning weights of 5.0, 3.0, and 2.0 causes approximately 50% of dispatched Tasks to come from `premium`, 30% from `basic`, and 20% from `free` when all three groups have backlogged Tasks. Within a Fairness key, Tasks at the same priority are dispatched in first-in-first-out (FIFO) order. @@ -54,17 +54,15 @@ See [Task Queue Priority and Fairness](/develop/task-queue-priority-fairness#tas ## When to use -This pattern is a good fit for multi-tenant applications where large tenants should not block small tenants. It also fits workloads that need proportional Task dispatch across groups without hard per-group limits, and applications where tenants or groups change often. New Fairness keys can be introduced without deploying new Workers. - -For a broader look at multi-tenancy strategies in Temporal, see [Multi-Tenant Patterns](/best-practices/multi-tenant-patterns). +This pattern is a good fit for multi-tenant applications where large tenants should not block small tenants, for workloads that need proportional Task dispatch across groups without hard rate limits, and when the set of tenants or groups is dynamic. New Fairness keys can be introduced without deploying new Workers. For a broader look at multi-tenancy strategies in Temporal, see [Multi-Tenant Patterns](/best-practices/multi-tenant-patterns). Fairness does not provide exact dispatch ratios, concurrency limits, or compute isolation. Use Activity Task Queue rate limits for throughput caps. Use separate Task Queues with dedicated Worker pools and compute resources for hard isolation. Use Priority to order urgent work ahead of less urgent work. ## Benefits and trade-offs -Fairness is work-conserving. A group can use all available dispatches when no other group has a backlog. New groups can start using the same Task Queue without changes to the Worker deployment. +A single Worker pool serves all tenants, so idle capacity from a low-traffic tenant is available to high-traffic tenants. New tenants require no Worker deployment. Add a Fairness key and Temporal starts dispatching their Tasks. Fairness weights can be updated through Task Queue configuration without redeploying application code. -Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, Worker Versioning, and short time windows. Tasks with different runtimes can consume different amounts of Worker capacity even when their dispatch shares match their weights. +Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, Worker Versioning, and short time windows. Accuracy can degrade with a large number of Fairness keys. Fairness weights apply when Tasks are scheduled, so changing a weight does not reorder Tasks already in the backlog. Tasks with different runtimes can consume different amounts of Worker capacity even when their dispatch shares match their weights. ## Comparison with alternatives @@ -77,8 +75,8 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, ## Best practices -- **Use stable Fairness keys.** Use account identifiers or tenant slugs instead of display names. -- **Combine Priority and Fairness for mixed workloads.** Use Priority for urgency classes and Fairness for tenants within each class. +- **Use stable, consistent Fairness keys.** Use account identifiers or tenant slugs instead of display names. Key changes do not reorder Tasks already in the backlog. +- **Combine Priority and Fairness for multi-class, multi-tenant workloads.** Priority separates urgent work from batch work. Fairness prevents a single tenant from dominating within each Priority level. ## Common pitfalls diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index f99f7e4318..9bec9092a3 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -5,24 +5,24 @@ description: "Orders Task dispatches so urgent work moves ahead of lower-priorit --- :::info[TLDR] -Assign a Priority key from 1 to 5 to Workflows, Activities, and Child Workflows. When Tasks are backlogged, Temporal dispatches higher-priority Tasks first. Use Priority when urgent work should move ahead of a backlog. +Assign a Priority key from 1 to 5 to Workflows, Activities, and Child Workflows so higher-priority Tasks dispatch ahead of lower-priority Tasks on a shared Task Queue. Use this when a flood of batch or background Tasks would otherwise delay urgent work. ::: ## Overview -Priority orders Task dispatches within a shared Task Queue. Lower Priority key values represent higher priority, so `1` is the highest priority and `5` is the lowest. +The Priority pattern orders Task dispatches within a shared Task Queue so that time-sensitive work can move ahead of lower-priority work without separate queues or routing logic. Lower Priority key values represent higher priority, so `1` is the highest priority and `5` is the lowest. Priority applies to dispatch. It does not preempt running Tasks or reserve Worker capacity. ## Problem -Without Priority, backlogged Tasks are generally dispatched in first-in-first-out (FIFO) order within a Task Queue partition. A large batch can fill the backlog before urgent work arrives, delaying time-sensitive Tasks. +In a shared Task Queue, backlogged Tasks are generally dispatched in first-in-first-out (FIFO) order within a partition. When a large batch of low-priority work, such as nightly reports, bulk imports, or background processing, fills the backlog before time-sensitive requests arrive, the urgent work waits behind the batch. -Separate Task Queues can isolate the backlogs, but require more routing and Worker configuration. +A Task Queue without Priority gives the same dispatch preference to all Tasks, regardless of business urgency. ## Solution -Assign a Priority key to each Workflow, Activity, or Child Workflow. Within a Task Queue partition, the Matching Service dispatches the highest-priority backlogged Tasks first. Without Fairness, Tasks with the same Priority key are dispatched in FIFO order. +Temporal's Priority feature lets you assign a Priority key from 1 to 5 to any Workflow, Activity, or Child Workflow. Within a Task Queue partition and Worker Deployment Version, the Matching Service maintains a sub-queue for each Priority level and dispatches the highest-priority backlogged Tasks first. Without Fairness, Tasks at the same Priority level are dispatched in FIFO order. Tasks use Priority key `3` by default. Activities and Child Workflows inherit the parent Workflow's Priority key unless they set their own. @@ -37,9 +37,15 @@ flowchart TD P1 -->|dispatch first| W["Shared Workers"] P3 -->|dispatch next| W P5 -->|dispatch last| W + W --> DS["Downstream\nService"] ``` -The diagram assumes all three levels have backlogged Tasks. Lower-priority Tasks wait for higher-priority backlogs to drain. +The diagram assumes all three levels have backlogged Tasks in the same partition and Worker Deployment Version. + +1. Workflows start with a Priority key in their start options. Payment Workflows use Priority `1`, routine Workflows use the default Priority `3`, and nightly batch reports use Priority `5`. +2. The Matching Service routes each Task to its Priority sub-queue within the Task Queue. +3. Workers poll the Task Queue and receive the highest-priority backlogged Tasks first. +4. Activities and Child Workflows inherit the parent Workflow's Priority key unless they set their own. ## Implementation @@ -57,7 +63,7 @@ Priority adds little when all work has the same urgency. A sustained high-priori ## Benefits and trade-offs -Priority keeps work on one Task Queue and Worker pool. Urgency levels require no separate routing or Workers. All levels share idle Worker capacity. When one level has no backlog, Tasks at other levels can use the available dispatches. +Native Priority requires no extra queues, routing logic, or additional Worker pools. A single pool of Workers serves all Priority levels, so idle Worker capacity is shared across all levels. Priority applies only to Tasks waiting for dispatch. It does not preempt Tasks that are already running. A sustained higher-priority backlog can delay lower-priority Tasks indefinitely. Priority supports five levels, from `1` for the highest priority to `5` for the lowest. @@ -72,8 +78,10 @@ Priority applies only to Tasks waiting for dispatch. It does not preempt Tasks t ## Best practices -- **Define a small set of levels.** For example, use `1` for urgent work, `3` for normal work, and `5` for batch work. -- **Use inheritance.** Set an Activity or Child Workflow's Priority key only when it should differ from its parent Workflow. +- **Keep Priority levels coarse.** For example, use `1` for urgent work, `3` for normal work, and `5` for batch work. +- **Reserve Priority `1` for urgent work.** When every caller uses the highest Priority, the feature provides no ordering benefit. +- **Set the initial Priority key in Workflow start options.** Activities and Child Workflows inherit it unless they set their own. +- **Override Activity Priority deliberately.** Use a different Priority key only when an Activity should dispatch at a different level than its Workflow. ## Common pitfalls From 79967a6fc498bdf19a8867f1a757bc93271db11a Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:35:08 -0700 Subject: [PATCH 16/26] Restore Fairness comparison table --- docs/design-patterns/fairness.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 3cc017b952..a2ec269b97 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -66,12 +66,12 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, ## Comparison with alternatives -| Approach | Backlog dispatch | Shares idle capacity | Tenant onboarding | -| :--- | :--- | :--- | :--- | -| Fairness on a shared Task Queue | Weighted across backlogged groups | Yes | Assign a Fairness key | -| Task Queue per tenant with shared compute | Separate tenant backlogs | Yes | Add Task Queue and Worker configuration | -| Task Queue per tenant with dedicated compute | Separate tenant backlogs | No | Deploy and configure dedicated Workers | -| Shared Task Queue without Fairness | No tenant-aware ordering | Yes | No additional configuration | +| Approach | Per-tenant dispatch | Dynamic tenants | Shares idle capacity | Complexity | +| :--- | :--- | :--- | :--- | :--- | +| Temporal Fairness (native) | Weighted | Yes | Yes | Low | +| Dedicated Task Queue per tenant | Separate | No | No | Medium | +| Single shared Task Queue (no control) | None | Yes | Yes | Lowest | +| External queue with per-tenant consumer groups | Separate | Yes | No | High | ## Best practices From 67330174897bd91a1407c354500b0964c83188ba Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:39:39 -0700 Subject: [PATCH 17/26] Preserve Priority pattern narrative --- docs/design-patterns/priority-task-queues.mdx | 30 ++++++++----------- 1 file changed, 12 insertions(+), 18 deletions(-) diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index 9bec9092a3..2008093643 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -5,26 +5,22 @@ description: "Orders Task dispatches so urgent work moves ahead of lower-priorit --- :::info[TLDR] -Assign a Priority key from 1 to 5 to Workflows, Activities, and Child Workflows so higher-priority Tasks dispatch ahead of lower-priority Tasks on a shared Task Queue. Use this when a flood of batch or background Tasks would otherwise delay urgent work. +Assign a Priority key from 1 to 5 to Workflows and Activities so high-priority work dispatches ahead of lower-priority work on a shared Task Queue. Use this when a flood of batch or background Tasks would otherwise delay high-urgency requests. ::: ## Overview -The Priority pattern orders Task dispatches within a shared Task Queue so that time-sensitive work can move ahead of lower-priority work without separate queues or routing logic. Lower Priority key values represent higher priority, so `1` is the highest priority and `5` is the lowest. +The Priority pattern assigns a Priority key to Workflows, Activities, and Child Workflows so that time-sensitive work dispatches ahead of lower-priority work within a single Task Queue, without requiring separate queues or routing logic. Priority applies to dispatch. It does not preempt running Tasks or reserve Worker capacity. ## Problem -In a shared Task Queue, backlogged Tasks are generally dispatched in first-in-first-out (FIFO) order within a partition. When a large batch of low-priority work, such as nightly reports, bulk imports, or background processing, fills the backlog before time-sensitive requests arrive, the urgent work waits behind the batch. - -A Task Queue without Priority gives the same dispatch preference to all Tasks, regardless of business urgency. +In a shared Task Queue, backlogged Tasks are generally dispatched in first-in-first-out (FIFO) order within a partition. When a large batch of low-priority work, such as nightly reports, bulk imports, or background processing, fills the backlog before time-sensitive requests arrive, the higher-priority requests wait behind the entire batch. A single Task Queue with no ordering mechanism gives the same dispatch preference to all Tasks, regardless of business urgency. ## Solution -Temporal's Priority feature lets you assign a Priority key from 1 to 5 to any Workflow, Activity, or Child Workflow. Within a Task Queue partition and Worker Deployment Version, the Matching Service maintains a sub-queue for each Priority level and dispatches the highest-priority backlogged Tasks first. Without Fairness, Tasks at the same Priority level are dispatched in FIFO order. - -Tasks use Priority key `3` by default. Activities and Child Workflows inherit the parent Workflow's Priority key unless they set their own. +Temporal's native Priority feature lets you assign a Priority key (an integer from 1 to 5, where 1 is the highest priority and 5 is the lowest) to any Workflow, Activity, or Child Workflow. The Matching Service maintains a sub-queue for each Priority level and exhausts all backlogged Tasks at a given level before dispatching to the next. Tasks default to Priority `3` when no key is set. Activities and Child Workflows inherit the parent Workflow's Priority unless they set their own. ```mermaid flowchart TD @@ -40,12 +36,12 @@ flowchart TD W --> DS["Downstream\nService"] ``` -The diagram assumes all three levels have backlogged Tasks in the same partition and Worker Deployment Version. +The diagram assumes all three levels have backlogged Tasks in the same Task Queue partition and Worker Deployment Version. -1. Workflows start with a Priority key in their start options. Payment Workflows use Priority `1`, routine Workflows use the default Priority `3`, and nightly batch reports use Priority `5`. -2. The Matching Service routes each Task to its Priority sub-queue within the Task Queue. +1. Workflows start with a Priority key in their start options. Payment Workflows use Priority `1`. Routine Workflows default to Priority `3`. Nightly batch reports use Priority `5`. +2. The Matching Service routes each Task to the corresponding Priority sub-queue inside the Task Queue. 3. Workers poll the Task Queue and receive the highest-priority backlogged Tasks first. -4. Activities and Child Workflows inherit the parent Workflow's Priority key unless they set their own. +4. Activities and Child Workflows inherit the parent Workflow's Priority unless they set their own. ## Implementation @@ -55,17 +51,15 @@ See [Task Queue Priority](/develop/task-queue-priority-fairness#task-queue-prior ## When to use -Use Priority when a Task Queue handles work with different levels of urgency. Common examples include payments or user-facing requests sharing Workers with reports, data imports, or inventory updates. When Tasks back up, Priority dispatches urgent Tasks ahead of routine Tasks. - -Priority also works well for exceptional Tasks that should dispatch ahead of normal work, such as an operator-triggered recovery Task. +This pattern is a good fit when your system mixes time-sensitive operations (payment processing, user-facing requests) with background or batch work (reporting, data imports, inventory management), and you want urgent Tasks to dispatch first during periods of high load. It also works well when you need to mark urgent Tasks that should dispatch ahead of normal processing, for example, triggering immediate reruns of failed critical Tasks. -Priority adds little when all work has the same urgency. A sustained high-priority backlog can starve lower-priority Tasks. Use separate Task Queues with dedicated Workers and compute for capacity isolation. Use [Fairness](/design-patterns/fairness) when tenants within a Priority level need weighted shares of dispatches. +It is not a good fit when all work is effectively equal in urgency, when a continuously replenished high-priority backlog could starve lower-priority work indefinitely, or when you need hard capacity isolation between tiers. Use separate Task Queues with dedicated Worker pools and compute resources for hard capacity isolation. If your concern is prioritizing work among tenants or customers, consider [Fairness](/design-patterns/fairness), which distributes dispatches proportionally using weighted Fairness keys. ## Benefits and trade-offs Native Priority requires no extra queues, routing logic, or additional Worker pools. A single pool of Workers serves all Priority levels, so idle Worker capacity is shared across all levels. -Priority applies only to Tasks waiting for dispatch. It does not preempt Tasks that are already running. A sustained higher-priority backlog can delay lower-priority Tasks indefinitely. Priority supports five levels, from `1` for the highest priority to `5` for the lowest. +Lower-priority Tasks wait while higher-priority Tasks remain backlogged. In an environment with a continuously replenished high-priority backlog, low-priority Tasks may be delayed indefinitely. The built-in Priority key range is 1 to 5. The feature does not support more than five levels. ## Comparison with alternatives @@ -78,7 +72,7 @@ Priority applies only to Tasks waiting for dispatch. It does not preempt Tasks t ## Best practices -- **Keep Priority levels coarse.** For example, use `1` for urgent work, `3` for normal work, and `5` for batch work. +- **Use no more than five Priority levels.** Keep levels coarse. For example, use `1` for urgent work, `3` for normal work, and `5` for batch work. - **Reserve Priority `1` for urgent work.** When every caller uses the highest Priority, the feature provides no ordering benefit. - **Set the initial Priority key in Workflow start options.** Activities and Child Workflows inherit it unless they set their own. - **Override Activity Priority deliberately.** Use a different Priority key only when an Activity should dispatch at a different level than its Workflow. From cdd2e6a85364843506f8ed96593a31fa16ab30d9 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:40:58 -0700 Subject: [PATCH 18/26] Leave canonical Priority and Fairness guide unchanged --- docs/develop/task-queue-priority-fairness.mdx | 35 ++++++++++--------- .../PriorityFairnessWalkthrough/HowItWorks.js | 6 ++-- .../PriorityFairnessWalkthrough/Overview.js | 2 +- 3 files changed, 22 insertions(+), 21 deletions(-) diff --git a/docs/develop/task-queue-priority-fairness.mdx b/docs/develop/task-queue-priority-fairness.mdx index bcb73ba896..5f06f9422b 100644 --- a/docs/develop/task-queue-priority-fairness.mdx +++ b/docs/develop/task-queue-priority-fairness.mdx @@ -210,16 +210,17 @@ await Workflow.ExecuteChildWorkflowAsync( Task Queue Fairness lets you distribute Tasks based on _fairness keys_ and _fairness weights_ within a Task Queue. -Fairness keys organize Tasks into groups such as tenants, applications, or workload types. Weighted fair dispatch selects among groups that have backlogged Tasks. This keeps a group that fills the backlog from dominating dispatch. +Each fairness key creates its own "virtual queue", allowing you to organize Tasks into logical groups like tenants, applications, or workload types. These virtual queues operate using a round-robin dispatch mechanism, meaning the system cycles through each fairness key in turn when selecting the next Task to dispatch. This prevents any single fairness key from hogging Worker capacity, even if one key has a much larger backlog than the others. -Each fairness key has a default _fairness weight_ of 1.0. You can assign a different weight to a key. When two groups have backlogged Tasks, a group with a weight of 2.0 receives approximately twice as many dispatches as a group with a weight of 1.0. +By default, each fairness key is weighted equally in the round-robin, with a _fairness weight_ of 1.0. This behavior can be customized by assigning a different fairness weight to a key. For example, Tasks belonging to a fairness key with a weight of 2.0 will be dispatched twice as often as keys with the default weight. ### When to use Fairness Fairness is intended to address common situations like: - Multi-tenant applications with big and small tenants where small tenants shouldn't be blocked by big ones. -- Assigning Tasks to weighted groups and dispatching approximately 80% from one group and 20% from another when both have backlogged Tasks. +- Assigning Tasks to different capacity bands and then, for example, dispatching 80% from one band and 20% from another + without limiting overall capacity when one band is empty. It sequences Tasks in the Task Queue probabilistically using a weighted distribution based on: @@ -230,7 +231,7 @@ It sequences Tasks in the Task Queue probabilistically using a weighted distribu As an example, imagine a workload with three tenants, _tenant-big_, _tenant-mid_, _tenant-small_, that have varying numbers of Tasks at all times. Your _tenant-big_ has a large number of Tasks that can overwhelm your Task Queue and prevent _tenant-mid_ and _tenant-small_ from running their Tasks. With Fairness, you can give each tenant a different -fairness key to make sure _tenant-big_ doesn't dominate dispatch and block the others. In this case, +fairness key to make sure _tenant-big_ doesn't use all of the Task Queue resources and block the others. In this case, _tenant-mid_ and _tenant-small_ will have Tasks run in between _tenant-big_ Tasks so that they are executed "fairly". ### How to use Fairness @@ -241,11 +242,11 @@ To enable Fairness for a Namespace in Temporal Cloud, navigate to the Namespace' If you're self-hosting Temporal, set `matching.enableFairness` to `true` in the [dynamic config](/temporal-service/configuration#dynamic-configuration) on the relevant Task Queues or Namespaces. -To use Fairness, set fairness keys and optionally fairness weights at the Workflow, Activity, or Child Workflow level. Tasks with different fairness keys are dispatched in proportion to their fairness weights. For example, weights of 5.0 for _premium-tier_, 3.0 for _basic-tier_, and 2.0 for _free-tier_ cause approximately 50% of dispatched Tasks to come from _premium-tier_, 30% from _basic-tier_, and 20% from _free-tier_ when all three groups have backlogged Tasks. Within the same priority level and Task Queue partition, Tasks in the backlog with the same fairness key are dispatched in [FIFO order](/task-queue#task-ordering). +To use Fairness, you need to set fairness keys and optionally fairness weights at the Workflow, Activity, or Child Workflow level. Tasks with different fairness keys are dispatched in proportion to their fairness weights. For example, if you weight _premium-tier_ at 5.0, _basic-tier_ at 3.0, and _free-tier_ at 2.0, then 50% of dispatched Tasks come from _premium-tier_, 30% from _basic-tier_, and 20% from _free-tier_. If there are Tasks in the Task Queue backlog that have the same fairness key, then they're dispatched in [FIFO order](/task-queue#task-ordering). You can set a Workflow's fairness key and weight via the CLI like so: @@ -652,7 +653,7 @@ var handle = await Client.StartWorkflowAsync( -Tasks that do not have a `fairness_key` set are grouped under an implicit empty-string key with a default weight of 1.0. The group participates in weighted fair dispatch alongside named fairness keys. This lets you adopt Fairness incrementally. +Tasks that do not have a `fairness_key` set are grouped together under an implicit empty-string key. All unkeyed Tasks share this single default bucket and participate in the same round-robin dispatch alongside named fairness keys, with a default weight of 1.0. This means Fairness adoption can be incremental: you can assign fairness keys to some tenants but not others. Unkeyed Tasks do not bypass Fairness; they compete as one group alongside all explicitly keyed Tasks. :::info @@ -663,20 +664,20 @@ There should only be one fairness weight assigned to each fairness key within a ### Choosing between Priority, Fairness, and both - **Priority alone** when you need strict priority ordering - for example, separating real-time Tasks from batch Tasks. -- **Fairness alone** when you need weighted dispatch among tiers or tenants without ordering one group ahead of another. +- **Fairness alone** when you need tier or tenant isolation so no group is starved, but you don't need to preempt any group ahead of another. - **Both** when you have a tiered SLA hierarchy - Priority for the broad tier (for example, paid vs. free), Fairness for per-tenant equity within a tier. When you use Priority and Fairness together, the next Task to dispatch is chosen by walking three rules in order: 1. **Priority tier (strict).** Tasks at a higher priority always dispatch before tasks at lower priorities, regardless of fairness keys or weights. -2. **Fairness key within a tier (weighted).** Within a priority tier, Tasks are dispatched according to the weights of their fairness keys. When both groups are backlogged, a key with weight 2.0 receives approximately twice as many dispatches as a key with weight 1.0. +2. **Fairness key within a tier (weighted).** Within a priority tier, each fairness key is a virtual queue. Keys are dispatched proportional to their weights - a key with weight 2.0 is dispatched twice as often as one with weight 1.0. 3. **FIFO within a key.** Tasks that share a priority tier _and_ fairness key dispatch in the order they were enqueued. These rules apply within a Task Queue partition. ### Inheritance @@ -729,11 +730,12 @@ temporal task-queue config set \ **Whole queue rate limits:** applies to the whole queue regardless of the fairness key. This is the same setting as is exposed through the [Worker Options](/develop/worker-tuning-reference#io-configuration-options) in the SDKs, and when set via the API, takes precedence over the limit set through Worker Options. -**Fairness key rate limits:** The per-fairness-key rate limit caps the dispatch rate for each fairness key. Some important notes on the per-fairness-key limit: +**Fairness key rate limits:** The per-fairness-key rate limit works in conjunction with Task Queue Fairness. If you think of Fairness as dividing the queue into one virtual queue for each key, then the per-fairness-key rate limit is a limit on each individual virtual queue. Some important notes on the per-fairness-key limit: - The whole queue limit and per-fairness-key limit may be set independently: none, one or the other, or both may be set. If both are set, then the more restrictive one applies. -- The per-fairness-key limit for a key is scaled by the fairness weight assigned to that key. If the default limit is 10 Tasks per second, a key with weight 1.0 has a limit of 10 Tasks per second and a key with weight 2.5 has a limit of 25 Tasks per second. -- When a Task would exceed its key's rate limit, matching can skip it and dispatch another eligible Task. +- The per-fairness-key limit for a key is scaled by the fairness weight assigned to that key. So if the per-fairness-key limit for a queue is set to 10, then all keys with the default weight (1.0) will have a limit of 10 tasks/second. But if a particular key is given a weight of 2.5, then the per-key rate limit for that key will be 25 tasks/second. +- Since the dispatch rate for each key should be proportional to its weight, if any key is hitting the per-key limit, then nearly all of them are. The way it works is if the next Task to be dispatched hits the per-key limit, then dispatch will wait until it can go. +- Usually there isn't actually any blocking, but there can be when the fairness weight for a key is changed between when a Task is scheduled and when it's dispatched. If the fairness weight for a key is lowered, for example, the new lower per-key rate limit will be respected. Since those Tasks were originally scheduled with the higher rate, they will block other Tasks as they're dispatched. This limitation will be improved in the future. ### Fairness weight overrides @@ -758,8 +760,7 @@ To unset a single key's override, pass `key=default`. To clear all overrides on - There isn't a limit on the number of fairness keys you can use, but their accuracy can degrade as you add more. - Fairness is enforced within a single Task Queue [partition](/task-queue#task-ordering). When a Task Queue's partitions are imbalanced, Fairness may not appear to hold, since it applies only within individual partitions. Depending on your use case, you can reach out to Temporal Support to get your Task Queues set to a single partition. -- A Task's fairness weight is recorded when the Task is scheduled. Changing a weight in application code affects newly scheduled Tasks, not the current backlog. -- With Worker Versioning, each version has a separate backlog. Fairness applies within each version's backlog. -- When a Task Queue partition reloads or changes ownership, Temporal restores fairness state for up to 100 keys by default. Other keys rebuild their state as new Tasks arrive, which can temporarily distort weighted dispatch. Fairness pass dithering spreads the initial positions of these keys according to their weights. It can reduce FIFO-like ordering among equal-weight keys. To enable fairness pass dithering, [contact Temporal Support](/cloud/support#support-ticket). +- The fairness weight applies at schedule time, not at dispatch time. So it only affects newly-scheduled Tasks, not currently backlogged ones. This means if you need to throttle a single fairness key in the existing backlog of Tasks, you won't be able to. +- When you use Worker Versioning and you're moving Workflows from one version to another, Priority will still apply between versions. Fairness isn't guaranteed between versions. For example, you may have Tasks that were originally queued on Worker version _alpha_, Tasks that were queued on Worker version _beta_, and some Tasks were moved from _alpha_ to _beta_. Fairness is only guaranteed when Tasks are originally queued on the same Worker version. So there might be some discrepancies on the Tasks moved from _alpha_ to _beta_. +- During server restarts, Temporal preserves fairness state for the top 100 keys. Other keys rebuild their fairness state as new Tasks arrive, which can temporarily distort weighted dispatch. Fairness pass dithering reduces this distortion by spreading keys' initial positions according to their weights. It can reduce FIFO-like ordering among equal-weight keys. To enable fairness pass dithering, [contact Temporal Support](/cloud/support#support-ticket). - Fairness doesn't consider Task executions that have already been dispatched to Workers. As a result, fair dispatch may not be immediately visible in the mix of Tasks currently running on Workers. -- Tasks that synchronously match an available poller can bypass backlog ordering. [Eagerly dispatched Tasks](/develop/worker-performance#eager-task-execution) also bypass matching. diff --git a/src/components/elements/PriorityFairnessWalkthrough/HowItWorks.js b/src/components/elements/PriorityFairnessWalkthrough/HowItWorks.js index 837e6019e4..48af60e667 100644 --- a/src/components/elements/PriorityFairnessWalkthrough/HowItWorks.js +++ b/src/components/elements/PriorityFairnessWalkthrough/HowItWorks.js @@ -8,8 +8,8 @@ const STEPS = [ body: <>Tasks at a higher priority (lower number) are always dispatched before tasks at a lower priority. Every task at priority 1 is dispatched before any task at priority 2, and so on., }, { - title: 'Fairness distributes dispatches within the tier', - body: <>Within a priority tier, weighted fair dispatch selects among fairness keys with backlogged Tasks. This prevents one fairness key from filling the backlog and dominating dispatch., + title: 'Fairness distributes capacity within the tier', + body: <>Within a priority tier, tasks are dispatched proportionally by fairnessWeight using a weighted round-robin mechanism. This prevents any single fairnessKey from hogging Worker capacity, even if it has a deep backlog., }, { title: <>No fairnessKey means FIFO within the tier, @@ -32,7 +32,7 @@ export default function HowItWorks({ onNext }) {

When a Worker polls for the next task, Temporal applies two rules in sequence: Priority - determines which tier goes first, and Fairness distributes dispatches among keys within each + determines which tier goes first, and Fairness distributes capacity among keys within each tier.

diff --git a/src/components/elements/PriorityFairnessWalkthrough/Overview.js b/src/components/elements/PriorityFairnessWalkthrough/Overview.js index 8424ac4869..dfc4528f6e 100644 --- a/src/components/elements/PriorityFairnessWalkthrough/Overview.js +++ b/src/components/elements/PriorityFairnessWalkthrough/Overview.js @@ -62,7 +62,7 @@ export default function Overview({ onNext }) {

Without Fairness, tasks at the same priority dispatch in FIFO order, so a backlog-heavy - tenant can fill the backlog and dominate dispatch at that level. Fairness groups + tenant can hog Worker capacity and delay everyone else at that level. Fairness groups tasks by a fairness key and dispatches them proportionally by fairness weight. A key with weight 2.0 is dispatched twice as often as a key with the default weight of 1.0.

From 6c687554f91e8e7113950a3020b4ed7e0b073a47 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:42:48 -0700 Subject: [PATCH 19/26] Keep virtual queue model in Fairness comparison --- docs/design-patterns/fairness.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index a2ec269b97..3d464339aa 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -68,7 +68,7 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, | Approach | Per-tenant dispatch | Dynamic tenants | Shares idle capacity | Complexity | | :--- | :--- | :--- | :--- | :--- | -| Temporal Fairness (native) | Weighted | Yes | Yes | Low | +| Temporal Fairness (native) | Weighted virtual queues | Yes | Yes | Low | | Dedicated Task Queue per tenant | Separate | No | No | Medium | | Single shared Task Queue (no control) | None | Yes | Yes | Lowest | | External queue with per-tenant consumer groups | Separate | Yes | No | High | From 72c06b7236694170ba6001a439d0d63c37510466 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:44:25 -0700 Subject: [PATCH 20/26] Label Fairness diagram virtual queues --- docs/design-patterns/fairness.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 3d464339aa..4382e3b889 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -31,9 +31,9 @@ flowchart TD WA["Workflow\nfairness_key=tenant-big\nweight=1.0"] --> TQ["my-task-queue"] WB["Workflow\nfairness_key=tenant-mid\nweight=1.0"] --> TQ WC["Workflow\nfairness_key=tenant-small\nweight=1.0"] --> TQ - TQ --> VQ1["Fairness key\ntenant-big"] - TQ --> VQ2["Fairness key\ntenant-mid"] - TQ --> VQ3["Fairness key\ntenant-small"] + TQ --> VQ1["Virtual queue\ntenant-big"] + TQ --> VQ2["Virtual queue\ntenant-mid"] + TQ --> VQ3["Virtual queue\ntenant-small"] VQ1 -->|weighted dispatch| W["Shared Workers"] VQ2 -->|weighted dispatch| W VQ3 -->|weighted dispatch| W From a253ccf39d07aaf0b409e34330255b450d3bf24a Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:46:50 -0700 Subject: [PATCH 21/26] Use weighted fair dispatch in comparison --- docs/design-patterns/fairness.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index 4382e3b889..a6cc2c4eac 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -68,7 +68,7 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, | Approach | Per-tenant dispatch | Dynamic tenants | Shares idle capacity | Complexity | | :--- | :--- | :--- | :--- | :--- | -| Temporal Fairness (native) | Weighted virtual queues | Yes | Yes | Low | +| Temporal Fairness (native) | Weighted fair dispatch | Yes | Yes | Low | | Dedicated Task Queue per tenant | Separate | No | No | Medium | | Single shared Task Queue (no control) | None | Yes | Yes | Lowest | | External queue with per-tenant consumer groups | Separate | Yes | No | High | From 1e13f007362430026280cb3f81c0f1d0483109fe Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:48:08 -0700 Subject: [PATCH 22/26] Preserve Priority pitfall guidance --- docs/design-patterns/priority-task-queues.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index 2008093643..52ac92bec2 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -79,12 +79,12 @@ Lower-priority Tasks wait while higher-priority Tasks remain backlogged. In an e ## Common pitfalls -- **Assigning Priority key `1` to all work.** Priority cannot order Tasks when they all use the same key. +- **Assigning Priority `1` to all work by default.** When every caller sets the highest Priority, the feature provides no ordering benefit. Establish an explicit policy for which work types qualify for each level. +- **Neglecting low-priority starvation.** Under sustained high load, Priority `5` Tasks may wait indefinitely. Use a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) on low-priority Activities to surface starvation as a visible failure. +- **Assuming hard isolation between Priority levels.** Priority controls dispatch order, not Worker capacity allocation. A Priority `5` Task may still occupy a Worker slot when a Priority `1` Task arrives. - **Expecting Priority across Task Queue partitions.** Each partition orders its backlog independently. - **Expecting Priority across Worker Deployment Versions.** Each version has a separate backlog. Priority applies within each version's backlog. -- **Expecting FIFO order within a Priority level when using Fairness.** Fairness controls dispatch within each Priority level. - **Expecting every Task to pass through priority dispatch.** Synchronous matching can send a Task directly to an idle poller. [Eager Task Execution](/develop/worker-performance#eager-task-execution) bypasses matching. -- **Ignoring lower-priority starvation.** A sustained higher-priority backlog can prevent lower-priority Tasks from dispatching. ## Related From 4ce386f6e99ae0dbe6aef1fd47cab59d82ff6e9a Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:52:09 -0700 Subject: [PATCH 23/26] Restore original Priority guidance --- docs/design-patterns/priority-task-queues.mdx | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index 52ac92bec2..a1fd7bb6e5 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -72,19 +72,18 @@ Lower-priority Tasks wait while higher-priority Tasks remain backlogged. In an e ## Best practices -- **Use no more than five Priority levels.** Keep levels coarse. For example, use `1` for urgent work, `3` for normal work, and `5` for batch work. -- **Reserve Priority `1` for urgent work.** When every caller uses the highest Priority, the feature provides no ordering benefit. -- **Set the initial Priority key in Workflow start options.** Activities and Child Workflows inherit it unless they set their own. -- **Override Activity Priority deliberately.** Use a different Priority key only when an Activity should dispatch at a different level than its Workflow. +- **Use no more than five Priority levels.** The Priority key range is 1 to 5. Keep levels coarse. For example, use `1` for urgent work, `3` for normal work, and `5` for batch work. +- **Reserve Priority `1` for genuinely urgent work.** When every caller uses Priority `1`, the highest level fills with routine work and the feature provides no benefit. The default is `3` when no key is set. +- **Set the initial Priority key at Workflow start.** Set the Priority in the start options before execution begins. Activities and Child Workflows inherit it unless they set their own. +- **Override Activity Priority deliberately.** Activities inherit the parent Workflow's Priority by default. Override it only when a specific Activity must dispatch at a different level than its Workflow. +- **Monitor queue depth per Priority level.** Sustained backlog growth at a level means Tasks are arriving faster than they are being dispatched. ## Common pitfalls - **Assigning Priority `1` to all work by default.** When every caller sets the highest Priority, the feature provides no ordering benefit. Establish an explicit policy for which work types qualify for each level. - **Neglecting low-priority starvation.** Under sustained high load, Priority `5` Tasks may wait indefinitely. Use a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) on low-priority Activities to surface starvation as a visible failure. +- **Assuming Priority cannot change after scheduling.** Use `UpdateWorkflowExecutionOptions` to change a Workflow Execution's Priority and `UpdateActivityOptions` to change a pending Activity's Priority. - **Assuming hard isolation between Priority levels.** Priority controls dispatch order, not Worker capacity allocation. A Priority `5` Task may still occupy a Worker slot when a Priority `1` Task arrives. -- **Expecting Priority across Task Queue partitions.** Each partition orders its backlog independently. -- **Expecting Priority across Worker Deployment Versions.** Each version has a separate backlog. Priority applies within each version's backlog. -- **Expecting every Task to pass through priority dispatch.** Synchronous matching can send a Task directly to an idle poller. [Eager Task Execution](/develop/worker-performance#eager-task-execution) bypasses matching. ## Related From ea4c72f88de87a8e8bdd98df0069a737c933a2fd Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:53:54 -0700 Subject: [PATCH 24/26] Keep Priority copy close to original --- docs/design-patterns/index.mdx | 2 +- docs/design-patterns/priority-task-queues.mdx | 18 +++++++++--------- .../qos-throughput-patterns.mdx | 6 +++--- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/design-patterns/index.mdx b/docs/design-patterns/index.mdx index fdd7fe5958..41b095f002 100644 --- a/docs/design-patterns/index.mdx +++ b/docs/design-patterns/index.mdx @@ -215,7 +215,7 @@ Having these patterns in your toolbox helps you solve recurring problems in a ba href: "/design-patterns/priority-task-queues", icon: "priority-task-queues-icon.svg", title: "Priority", - description: "Orders Task dispatches so urgent work moves ahead of lower-priority work on the same Task Queue.", + description: "Assigns a priority level to Workflows and Activities so that time-sensitive work dispatches ahead of lower-priority work within a single Task Queue.", }, { href: "/design-patterns/fairness", diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index a1fd7bb6e5..5498c308ac 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -1,11 +1,11 @@ --- id: priority-task-queues title: "Priority" -description: "Orders Task dispatches so urgent work moves ahead of lower-priority work on the same Task Queue." +description: "Assigns a priority level to Workflows and Activities so that time-sensitive work dispatches ahead of lower-priority work within a single Task Queue." --- :::info[TLDR] -Assign a Priority key from 1 to 5 to Workflows and Activities so high-priority work dispatches ahead of lower-priority work on a shared Task Queue. Use this when a flood of batch or background Tasks would otherwise delay high-urgency requests. +Assign a Priority key from 1 to 5 to Workflows and Activities so **high-priority work dispatches ahead of lower-priority work** on a shared Task Queue. Use this when a flood of batch or background Tasks would otherwise delay high-urgency requests. ::: ## Overview @@ -30,13 +30,13 @@ flowchart TD TQ --> P1["Priority 1\nsub-queue"] TQ --> P3["Priority 3\nsub-queue"] TQ --> P5["Priority 5\nsub-queue"] - P1 -->|dispatch first| W["Shared Workers"] - P3 -->|dispatch next| W - P5 -->|dispatch last| W + P1 -->|dispatched first| W["Shared Workers"] + P3 -->|dispatched second| W + P5 -->|dispatched last| W W --> DS["Downstream\nService"] ``` -The diagram assumes all three levels have backlogged Tasks in the same Task Queue partition and Worker Deployment Version. +The following describes each step in the diagram. The example assumes all three levels have backlogged Tasks in the same Task Queue partition and Worker Deployment Version. 1. Workflows start with a Priority key in their start options. Payment Workflows use Priority `1`. Routine Workflows default to Priority `3`. Nightly batch reports use Priority `5`. 2. The Matching Service routes each Task to the corresponding Priority sub-queue inside the Task Queue. @@ -53,13 +53,13 @@ See [Task Queue Priority](/develop/task-queue-priority-fairness#task-queue-prior This pattern is a good fit when your system mixes time-sensitive operations (payment processing, user-facing requests) with background or batch work (reporting, data imports, inventory management), and you want urgent Tasks to dispatch first during periods of high load. It also works well when you need to mark urgent Tasks that should dispatch ahead of normal processing, for example, triggering immediate reruns of failed critical Tasks. -It is not a good fit when all work is effectively equal in urgency, when a continuously replenished high-priority backlog could starve lower-priority work indefinitely, or when you need hard capacity isolation between tiers. Use separate Task Queues with dedicated Worker pools and compute resources for hard capacity isolation. If your concern is prioritizing work among tenants or customers, consider [Fairness](/design-patterns/fairness), which distributes dispatches proportionally using weighted Fairness keys. +It is not a good fit when all work is effectively equal in urgency, when a continuously replenished high-priority backlog could starve lower-priority work indefinitely, or when you need hard capacity isolation between tiers. Use separate Task Queues with dedicated Worker pools and compute resources for hard capacity isolation. If your concern is prioritizing work among tenants or customers, consider the [Fairness](/design-patterns/fairness) pattern, which distributes dispatches proportionally using weighted Fairness keys rather than strict ordering. ## Benefits and trade-offs -Native Priority requires no extra queues, routing logic, or additional Worker pools. A single pool of Workers serves all Priority levels, so idle Worker capacity is shared across all levels. +Native Priority requires no extra queues, routing logic, or additional Worker pools. A single pool of Workers serves all Priority levels, so idle Worker capacity is available to every level without additional configuration. -Lower-priority Tasks wait while higher-priority Tasks remain backlogged. In an environment with a continuously replenished high-priority backlog, low-priority Tasks may be delayed indefinitely. The built-in Priority key range is 1 to 5. The feature does not support more than five levels. +Lower-priority Tasks are blocked while higher-priority Tasks remain backlogged. In an environment with a continuously replenished high-priority backlog, low-priority Tasks may be significantly delayed. The built-in Priority key range is 1 to 5. The feature does not support more than five levels. ## Comparison with alternatives diff --git a/docs/design-patterns/qos-throughput-patterns.mdx b/docs/design-patterns/qos-throughput-patterns.mdx index e06f3597b2..95be00b800 100644 --- a/docs/design-patterns/qos-throughput-patterns.mdx +++ b/docs/design-patterns/qos-throughput-patterns.mdx @@ -1,12 +1,12 @@ --- id: qos-throughput-patterns title: "QoS & Throughput Patterns" -description: "Pattern selection guide for controlling execution rate, protecting downstream services from overload, and distributing dispatch across tenants." +description: "Pattern selection guide for controlling execution rate, protecting downstream services from overload, and providing fair dispatch across tenants." --- import PatternCards from '@site/src/components/PatternCards'; -These patterns control how fast work executes, protect downstream services from overload, and keep one caller or tenant from dominating dispatch. +These patterns control how fast work executes, protect downstream services from overload, and make sure no single caller or tenant dominates dispatch. ## Patterns in this section @@ -21,7 +21,7 @@ These patterns control how fast work executes, protect downstream services from href: "/design-patterns/priority-task-queues", icon: "priority-task-queues-icon.svg", title: "Priority", - description: "Orders Task dispatches so urgent work moves ahead of lower-priority work on the same Task Queue.", + description: "Assigns a priority level to Workflows and Activities so time-sensitive work dispatches ahead of lower-priority work on the same Task Queue.", }, { href: "/design-patterns/fairness", From 62dac134a1156e27f3c9477500be81a206733962 Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:55:24 -0700 Subject: [PATCH 25/26] Restore queued Task priority guidance --- docs/design-patterns/priority-task-queues.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-patterns/priority-task-queues.mdx b/docs/design-patterns/priority-task-queues.mdx index 5498c308ac..d3bd7e638b 100644 --- a/docs/design-patterns/priority-task-queues.mdx +++ b/docs/design-patterns/priority-task-queues.mdx @@ -82,7 +82,7 @@ Lower-priority Tasks are blocked while higher-priority Tasks remain backlogged. - **Assigning Priority `1` to all work by default.** When every caller sets the highest Priority, the feature provides no ordering benefit. Establish an explicit policy for which work types qualify for each level. - **Neglecting low-priority starvation.** Under sustained high load, Priority `5` Tasks may wait indefinitely. Use a [Schedule-To-Start Timeout](/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) on low-priority Activities to surface starvation as a visible failure. -- **Assuming Priority cannot change after scheduling.** Use `UpdateWorkflowExecutionOptions` to change a Workflow Execution's Priority and `UpdateActivityOptions` to change a pending Activity's Priority. +- **Changing priority after scheduling.** The Priority key is evaluated when a Task enters the queue and cannot be changed while it waits. To re-prioritize an already-queued Task, cancel it and reschedule with the new priority. - **Assuming hard isolation between Priority levels.** Priority controls dispatch order, not Worker capacity allocation. A Priority `5` Task may still occupy a Worker slot when a Priority `1` Task arrives. ## Related From 1088365dd18c164a0076a67f52fc1a994060d5dc Mon Sep 17 00:00:00 2001 From: John Votta Date: Tue, 8 Sep 2026 20:58:34 -0700 Subject: [PATCH 26/26] Restore original Fairness pitfalls --- docs/design-patterns/fairness.mdx | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/design-patterns/fairness.mdx b/docs/design-patterns/fairness.mdx index a6cc2c4eac..85b1274f60 100644 --- a/docs/design-patterns/fairness.mdx +++ b/docs/design-patterns/fairness.mdx @@ -80,15 +80,13 @@ Fairness is best effort. Dispatch ratios can vary across Task Queue partitions, ## Common pitfalls -- **Expecting exact ratios.** Weights shape dispatch proportions over time when multiple groups have backlogs. Results vary across partitions and short time windows. -- **Expecting Fairness to reorder the backlog.** Temporal records each Task's Fairness weight when the Task is scheduled. Enabling Fairness or changing weights affects only newly scheduled Tasks. -- **Expecting Tasks without a Fairness key to bypass Fairness.** These Tasks share an implicit empty-string key with a weight of 1.0. -- **Expecting Fairness across Task Queue partitions.** Each partition calculates Fairness independently. Imbalanced partitions can change overall dispatch proportions. -- **Expecting Fairness across Worker Deployment Versions.** Each version has a separate backlog. Fairness applies within each version's backlog. -- **Expecting Fairness state to be fully restored after a server restart.** By default, Temporal restores state for up to 100 Fairness keys. Other keys rebuild state as Tasks arrive. -- **Treating Fairness as Worker capacity control.** Fairness considers only dispatch. A group with longer-running Tasks can consume more Worker time than its dispatch share suggests. +- **Expecting Fairness to reorder the existing backlog.** Fairness weight is evaluated at schedule time. When Fairness is enabled for a Namespace with an existing backlog, that backlog drains in its original order before fairness-aware dispatch applies to new Tasks. - **Using Fairness as a hard rate limiter.** Fairness by itself doesn't cap throughput. Use [Fairness key rate limits](/develop/task-queue-priority-fairness#set-rate-limits-at-the-task-queue-level) for per-key limits. -- **Expecting every Task to pass through fair dispatch.** Synchronous matching can send a Task directly to an idle poller. [Eager Task Execution](/develop/worker-performance#eager-task-execution) bypasses matching. +- **Unkeyed Tasks bypassing Fairness.** Tasks without a Fairness key are grouped under an implicit empty-string key and participate in weighted fair dispatch alongside named Fairness keys with a default weight of 1.0. They do not bypass Fairness and compete as one group. +- **Task Queue partitioning reducing accuracy.** Task Queues are internally partitioned, and Tasks are distributed to partitions randomly. This can interfere with fair dispatch proportions. If your workload requires higher accuracy, contact Temporal Support to configure a single-partition Task Queue. +- **Assuming Fairness applies across Worker Versioning boundaries.** Worker Deployment Versions have separate backlogs. Fairness applies within each version's backlog. +- **Expecting consistent Fairness immediately after a server restart.** Fairness ordering is preserved across restarts for the most active keys. Less active keys may briefly dispatch new Tasks ahead of their existing backlog until ordering normalizes. +- **Expecting the running Task mix to immediately reflect fair dispatch.** Fairness governs which Task is dispatched next. It does not account for Tasks already running on Workers. A group with longer-running Tasks can consume more Worker time than its dispatch share suggests. ## Related