Support active/passive failover between backend services

[josh-pritchard]

Description

Add support for active/passive (primary/fallback) failover between backend services. Users should be able to designate one backend as primary and another as a fallback that only receives traffic when the primary's health degrades, with automatic recovery back to the primary when it becomes healthy again.
This has been requested in #13507.

Motivation

Currently, when an HTTPRoute references multiple backendRefs, they are translated as weighted clusters — traffic is split proportionally. There is no way to express "only use backend B if backend A is unhealthy."

Proposed Approach

Use Envoy's priority-based load balancing. Endpoints from both the primary and fallback backends would be placed in the same Envoy cluster but in different LocalityLbEndpoints groups with different priority levels:

• Primary backend endpoints → priority: 0
• Fallback backend endpoints → priority: 1
  Envoy's built-in priority load balancing handles the rest: traffic goes to priority-0 endpoints first, and only spills over to priority-1 when the health of priority-0 drops below a threshold (controlled by the overprovisioning factor — at the default of 1.4, failover triggers when active health drops below ~72%).
  Health checks (active or passive via outlier detection) are required for failover detection and automatic recovery.

API Options

A few options for how to expose this in the kgateway API:

1. Annotation or field on backendRefs: e.g., a fallback: true annotation or a new field on the backendRef to indicate it is a fallback destination. This works for Backends and kube resources
2. New policy CRD field: e.g., a field on TrafficPolicy or BackendConfigPolicy that designates failover relationships between services. Policy can apply to Backends and kube resources
3. Extension to the Backend CRD: A fallback field on the Backend resource itself. Problem with this is we could need to add the kube type to the backend for this to work and it would change the UX users already have if they are directly referenced in routes.

Considerations

• Weighted routing and failover are mutually exclusive semantics for the same backendRefs list — the API should make this clear.
• Both primary and fallback backends share the same Envoy cluster, so they share cluster-level settings (timeouts, circuit breakers, LB algorithm). This is a limitation of the priority-based approach & we should make sure our choice of API respects this as well.
• Users should be strongly encouraged to configure health checks or outlier detection alongside failover, otherwise there's no mechanism to detect backend failure.
• The Gateway API spec does not currently have a standard for this pattern, so this would be a kgateway extension. Maybe a good topic for GEP?

[NomadXD]

This is a really useful and much needed feature. Can I work on this?
/assign

cc @puertomontt

[andy-fong]

• Annotation is probably not flexible enough because many times, a failover cluster could potentially be a primary cluster for other traffic and vice versa.
• BackendRef is from GW API, can we even add field to it? Otherwise, I like adding weight and priority fields to it. I think that will be cleanest.

• BackendConfigPolicy seems odd to me as it normally apply to backend without knowing the exact backend reference, but to define a relationship, you need specific backend reference that doesn't seem to fit.
• Not completely sure if it fits into TrafficPolicy either. Things in TrafficPolicy usually are not related to backend.

• Maybe a new "BackendType", currently we have AWS, Static, DynamicForwardProxy, GCP. Maybe a new one called PriorityGroup that allow you to put different backendRef in different priority groups but not sure if this the the one you mentioned you don't really like?

[NomadXD]

Thanks for the suggestions! Based on the feedback, adding a fallback field directly on the Backend CRD is not a good fit because:

• It would force users who currently reference Kubernetes Services directly in backendRefs to create Backend wrappers just to mark them as fallback, changing the existing UX.
• As @andy-fong pointed out, a failover backend could be a primary backend for other traffic, hence a global fallback flag on the Backend doesn't support this.

Also as @andy-fong mentioned, we can consider the approach of having a new BackendType like PriorityGroup and referring that in the HTTPRoute backendRefs as follows. Also we would need to create a BackendConfigPolicy and configure health checks for the backends in the PriorityGroup for active/passive failover to work.

apiVersion: gateway.kgateway.dev/v1alpha1
kind: Backend
metadata:
  name: my-failover
spec:
  priorityGroup:
    primary:
    - kind: Service
      name: primary-svc
      port: 8080
    fallback:
    - kind: Service
      name: fallback-svc
      port: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-httproute
spec:
  rules:
  - backendRefs:
    - group: gateway.kgateway.dev
       kind: Backend
       name: my-failover

For simplicity, the initial API uses primary and fallback fields which map to Envoy priority levels 0 and 1 respectively. This keeps the API clean for the common active/passive use cases. However, Envoy natively supports multiple priority levels, which enables chained failover scenarios (e.g., same-region -> cross-region -> DR site).

If we start with primary/fallback and later need multi level failover, we'd have to introduce a separate priority based API as follows.

apiVersion: gateway.kgateway.dev/v1alpha1
kind: BackendGroup
metadata:
  name: my-failover
spec:
  priorityGroups:
  - priority: 0   # primary
    backends:
    - kind: Service
      name: primary-svc
      port: 8080
  - priority: 1   # fallback
    backends:
    - kind: Service
      name: fallback-svc
      port: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-httproute
spec:
  rules:
  - backendRefs:
    - group: gateway.kgateway.dev
      kind: BackendGroup
      name: my-failover

Alternatively, we could use explicit priority levels from the start. Slightly more verbose for the simple case but naturally extensible. Thoughts on which direction makes more sense here? I'm more biased towards the primary/fallback API to support one level of failover as the multi level failover scenario would be a niche use case most people don't require.

[NomadXD]

BackendRef is from GW API, can we even add field to it? Otherwise, I like adding weight and priority fields to it. I think that will be cleanest.

Yes, we cannot add a field to it as it from the K8s Gateway API and currently there's only a reference to the backend and an optional weight field. Ref: https://github.com/kubernetes-sigs/gateway-api/blob/c24ef843c6c67f10668f9cb1f830cfe481bdc8a7/apis/v1/shared_types.go#L302

I also think having this priority information in the backendRefs itself would be the cleanest approach similar to weights. We could probably open up a new GEP for this.

[andy-fong]

It's always hard to choose between flexibility vs simple UX. If we do this as you suggest:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: Backend
metadata:
  name: my-failover
spec:
  priorityGroup:
    primary:
    - kind: Service
      name: primary-svc
      port: 8080
    fallback:
    - kind: Service
      name: fallback-svc
      port: 8080

This definitely have a simple UX but I would not call it priorityGroup at all because it doesn't really fit. @josh-pritchard what do you think?

[NomadXD]

Also there are discussions related to this on the Gateway API project although there's no clear consensus on the direction.

kubernetes-sigs/gateway-api#2304
kubernetes-sigs/gateway-api#1802

[puertomontt]

Discussion:

consider using BackendConfigPolicy

consider zone aware routing as followup /related issue