From c5556648b41608e2d1d1f33d45523b0db0bd6b23 Mon Sep 17 00:00:00 2001 From: Tamal Saha Date: Wed, 1 Jul 2026 08:31:13 +0600 Subject: [PATCH] Add Kafka cross data center disaster recovery (DC-DR) docs Signed-off-by: Tamal Saha --- docs/guides/kafka/dr/_index.md | 10 + docs/guides/kafka/dr/guide/index.md | 360 +++++++++++++++++++++++++ docs/guides/kafka/dr/overview/index.md | 307 +++++++++++++++++++++ docs/guides/kafka/dr/runbook/index.md | 342 +++++++++++++++++++++++ 4 files changed, 1019 insertions(+) create mode 100644 docs/guides/kafka/dr/_index.md create mode 100644 docs/guides/kafka/dr/guide/index.md create mode 100644 docs/guides/kafka/dr/overview/index.md create mode 100644 docs/guides/kafka/dr/runbook/index.md diff --git a/docs/guides/kafka/dr/_index.md b/docs/guides/kafka/dr/_index.md new file mode 100644 index 000000000..fb4636cb9 --- /dev/null +++ b/docs/guides/kafka/dr/_index.md @@ -0,0 +1,10 @@ +--- +title: Disaster Recovery +menu: + docs_{{ .version }}: + identifier: kf-dr-kafka + name: DR + parent: kf-kafka-guides + weight: 36 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/kafka/dr/guide/index.md b/docs/guides/kafka/dr/guide/index.md new file mode 100644 index 000000000..a23860741 --- /dev/null +++ b/docs/guides/kafka/dr/guide/index.md @@ -0,0 +1,360 @@ +--- +title: DC-DR User Guide +menu: + docs_{{ .version }}: + identifier: kf-dr-guide-kafka + name: User Guide + parent: kf-dr-kafka + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Running Kafka in DC-DR Mode: User Guide + +This guide covers every aspect of operating a distributed Kafka in cross data center +disaster recovery (DC-DR) mode: the components, the naming contract, deployment, what +the operator creates, producing to the active endpoint, consumers resuming after a +flip, monitoring MM2 lag, the produce fence, switchover, failback, scaling, and day-2 +operations. + +Read the [DC-DR Overview](/docs/guides/kafka/dr/overview/index.md) first for the +architecture, and the [DC-DR Runbook](/docs/guides/kafka/dr/runbook/index.md) for +scenario-by-scenario procedures. + +> **New to KubeDB?** Please start [here](/docs/README.md). + +## Components and where they run + +| Component | Runs in | Responsibility | +| --- | --- | --- | +| **`dr-controlplane`** + 3-site etcd quorum | across the data centers (an OCM control plane) | Publishes one `coordination.k8s.io` **Lease** per failover scope. The Lease holder is the active write DC. This is the single cross-DC failover authority. | +| **`dr-controlplane` agent** | each spoke (DC) | Contends for the primary-DC Lease for its DC and projects the Lease decision into the local spoke as the primary-dc marker the produce fence reads. | +| **KubeDB Kafka operator (hub)** | the OCM hub | Expands the `Kafka` CR into per-DC Kafka clusters, per-DC `ConnectCluster` objects, and the MM2 `Connector` objects on the standby. On a Lease change it flips the bootstrap endpoint, reverses the MM2 direction, and moves the produce fence, then writes `status.disasterRecovery`. | +| **Per-DC Kafka clusters** | each Member DC | Each is a self-contained Kafka with its own intra-DC KRaft quorum, brokers, and per-partition ISR. KRaft never crosses the DC boundary. | +| **Per-DC `ConnectCluster` + MM2 connectors** | each Member DC | One `ConnectCluster` per DC (`kafkaRef` to its local Kafka). The three MM2 connectors run only on the current standby's `ConnectCluster`, mirroring active to standby. | +| **The produce fence** | each Member DC | Denies producer writes on a non-active cluster, fail closed, so only the Lease holder takes writes. | +| **KubeSlice (or external listeners)** | each spoke | Provides the flat cross-DC pod network so MM2 can reach the remote cluster and the bootstrap endpoint resolves across DCs. | + +## The DC-name contract + +One string identifies a data center everywhere. **Keep these identical:** + +- the OCM spoke cluster name +- the agent `--dc-name` +- the primary-DC Lease `holderIdentity` +- the marker `data.activeDC` +- the pod label `open-cluster-management.io/cluster-name` +- the `PlacementPolicy` `distributionRule.clusterName` + +## Deploying + +### PlacementPolicy + +Map the global pod ordinals to data centers and tag each DC with its role: + +```yaml +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + name: kf-dcdr +spec: + clusterSpreadConstraint: + slice: + projectNamespace: kubeslice-demo + sliceName: demo-slice + failoverPolicy: + mode: TwoDC # two Member DCs plus an Arbiter DC + trigger: + scope: Global # one cluster-wide failover scope + distributionRules: + - clusterName: dc-a + role: Member + replicaIndices: [0, 1, 2] + - clusterName: dc-b + role: Member + replicaIndices: [3, 4, 5] + - clusterName: dc-c + role: Arbiter + replicaIndices: [] +``` + +- A data-bearing **Member** rule carries `replicaIndices` that map to a self-contained + Kafka cluster with its own KRaft quorum. The **Arbiter** DC carries an empty + `replicaIndices` and holds only the `dr-controlplane` etcd member, never Kafka. +- `mode: TwoDC` expects exactly two Member DCs plus the Arbiter DC. Three or more data + DCs is a separate design and out of scope. +- Roles are `Member` and `Arbiter` only. + +### Kafka + +```yaml +apiVersion: kubedb.com/v1 +kind: Kafka +metadata: + name: kf-dcdr + namespace: demo +spec: + version: 4.0.0 + distributed: true + replicas: 6 + storageType: Durable + podTemplate: + spec: + podPlacementPolicy: + name: kf-dcdr + storage: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +### What the operator creates + +- **One self-contained Kafka cluster per Member DC** (`kf-dcdr` materialized into a + cluster in `dc-a` and a cluster in `dc-b`), each with its own KRaft controller + quorum and its own per-partition ISR. The two clusters never share a KRaft quorum. + `spec.replicas: 6` is the total broker count across both Member DCs; the + `replicaIndices` split it into a three-node cluster per DC (ordinals 0, 1, 2 in `dc-a` + and 3, 4, 5 in `dc-b`). +- **One `ConnectCluster` per DC**, each `kafkaRef` pointing at that DC's local Kafka, + because Connect's internal config, offset, and status topics must live on the + cluster that is the mirror target. The reverse-direction `ConnectCluster` stays + provisioned with its connectors disabled until a failover needs it. +- **The three MM2 `Connector` objects on the current standby's `ConnectCluster`**: a + `MirrorSourceConnector`, a `MirrorCheckpointConnector`, and a + `MirrorHeartbeatConnector`, mirroring the active cluster into the standby with + `IdentityReplicationPolicy`. +- **The Lease-gated bootstrap Service** that resolves to the active cluster's brokers. + +The MM2 connectors are ordinary KubeDB `Connector` objects. Conceptually the +`ConnectCluster` on the standby DC (`dc-b` while `dc-a` is active) looks like: + +```yaml +apiVersion: kafka.kubedb.com/v1alpha1 +kind: ConnectCluster +metadata: + name: kf-dcdr-dc-b-mm2 + namespace: demo +spec: + version: 4.0.0 + replicas: 3 + kafkaRef: + name: kf-dcdr-dc-b # the local (standby) Kafka: consume from remote, produce to local + namespace: demo + deletionPolicy: WipeOut +``` + +and the `MirrorSourceConnector` carries the mirror direction and the DR-critical +properties in its config secret: + +```yaml +apiVersion: kafka.kubedb.com/v1alpha1 +kind: Connector +metadata: + name: kf-dcdr-mirror-source + namespace: demo +spec: + configuration: + secretName: kf-dcdr-mirror-source-config + connectClusterRef: + name: kf-dcdr-dc-b-mm2 + namespace: demo + deletionPolicy: WipeOut +``` + +with the connector's `config.properties` (in the referenced secret) setting, among +others: + +```properties +connector.class=org.apache.kafka.connect.mirror.MirrorSourceConnector +source.cluster.alias=dc-a +target.cluster.alias=dc-b +source.cluster.bootstrap.servers=kf-dcdr-dc-a-pods.demo.svc:9092 +target.cluster.bootstrap.servers=kf-dcdr-dc-b-pods.demo.svc:9092 +replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy +sync.topic.acls.enabled=false +``` + +The operator owns these connector configs so the mirror direction stays +operator-controlled and the two directions never overlap. + +## Connecting and producing + +A DC-DR Kafka exposes one user-facing **bootstrap Service** that resolves to the +active cluster's brokers. Producers and consumers always connect to that single +endpoint and reach the write cluster without reconfiguration. Because MM2 uses +`IdentityReplicationPolicy`, topic names are identical on both clusters, so after the +endpoint flips clients keep using the same topics. + +```bash +# Produce to the active cluster through the single bootstrap endpoint: +$ kubectl exec -n demo kf-dcdr-dc-a-0 -- \ + kafka-console-producer.sh --bootstrap-server kf-dcdr-pods.demo.svc:9092 --topic orders +``` + +Only the active cluster accepts producer writes. If clients somehow reach a standby +cluster, its produce fence rejects the write (see below), which is the split-brain +guard. + +### Consumers resume after a flip + +The `MirrorCheckpointConnector` runs with `sync.group.offsets.enabled=true`, so it +translates consumer-group offsets from the active cluster into the standby cluster's +offset space. After a failover or switchover, a consumer group reconnecting through the +flipped endpoint resumes from the translated offset on the new active cluster rather +than re-reading from the beginning or skipping ahead. + +## Monitoring and observability + +### status.disasterRecovery + +The single CR carries the whole cross-DC view: + +```bash +$ kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery}' | jq +``` + +| Field | Meaning | +| --- | --- | +| `activeDC` | The DC that holds the Lease and takes producer writes. | +| `phase` | `Steady`, `FailingOver`, `FailingBack`, or `Degraded`. | +| `lastTransitionTime` | When `activeDC` last changed. | +| `dataCenters[].clusterName` | The data center, by its OCM managed cluster name. | +| `dataCenters[].role` | `Member` or `Arbiter`. | +| `dataCenters[].writable` | True only for the active cluster. | +| `dataCenters[].brokersReady` | Ready broker count in that DC's cluster. | +| `dataCenters[].mirrorLagMillis` | The standby's MM2 replication latency behind the active, in milliseconds. | +| `dataCenters[].healthy` | DC health: a Member DC is healthy when its brokers are ready; the Arbiter DC is healthy when its `dr-controlplane` etcd member is reachable (so an Arbiter reports `healthy: true` with `brokersReady: 0`). | + +### MM2 lag + +Cross-DC lag comes from MM2's own metrics: `replication-latency-ms` and +`record-age-ms` in the `kafka.connect.mirror` metric group, or the offset gap between +source and target derived from the heartbeat and checkpoint topics. The hub surfaces +this into `mirrorLagMillis`; there is no lag field in the base `KafkaStatus`. + +### Useful checks + +```bash +# Which DC the Lease intends as active (from the coordination plane): +$ kubectl --kubeconfig -n dc-failover get lease primary-dc \ + -o jsonpath='{.spec.holderIdentity}' + +# Per-DC brokers and roles: +$ kubectl get pods -n demo -l app.kubernetes.io/instance=kf-dcdr \ + -L kubedb.com/role,open-cluster-management.io/cluster-name + +# MM2 connector status on the standby's ConnectCluster: +$ kubectl get connector -n demo -l app.kubernetes.io/instance=kf-dcdr +``` + +## The produce fence + +A non-active cluster must refuse producer writes, fail closed: a cluster that cannot +confirm it holds the Lease denies produce. There are two fence mechanisms. + +- **ACL fence (authorization on):** revoke produce ACLs for the CLIENT principals on + the non-active cluster. This requires Kafka authorization enabled (the + `StandardAuthorizer`, set only when security is on). +- **Listener-gate fence (default posture):** gate the client listener on the non-active + cluster so producers cannot connect. The shipped examples default to + `disableSecurity: true`, and a cluster with no authorization has no ACLs to revoke, + so the listener gate is the default-posture fence. + +DC-DR therefore requires one of the two: auth-on for the ACL fence, or the listener +gate. Two rules keep the fence from breaking replication: + +- **Never fence the MM2 connector principal or `super.users`.** The fence must target + only client principals. Fencing the connector principal (or the operator's auth-secret + `super.users` user) would break mirroring and consumer-offset sync along with client + produce. +- **Do not blanket-mirror ACLs.** `MirrorSourceConnector` defaults + `sync.topic.acls.enabled=true`, which would copy the active cluster's client produce + ACLs onto the standby and re-grant produce there, undoing the fence. The operator sets + `sync.topic.acls.enabled=false` and manages client ACLs per cluster. + +## Planned switchover (drained, zero record loss) + +Move the active DC on purpose by annotating the Kafka: + +```bash +$ kubectl annotate kafka -n demo kf-dcdr dr.kubedb.com/switchover-to=dc-b +``` + +The hub then: + +1. checks the target is a known, healthy DC within the MM2 lag budget; +2. sets `phase: FailingOver` and quiesces producers by closing the active cluster's + produce fence; +3. waits for MM2 to drain to near-zero lag, so the target holds every record; +4. flips the bootstrap endpoint to the target, opens the target's fence, and reverses + the mirror direction (disables the old direction's connectors, then enables the new + direction's on the other DC's `ConnectCluster`), never both at once; +5. moves the Lease to the target. + +Because MM2 fully drained before the flip, no committed record is lost. This is a +hub-driven annotation, not a `KafkaOpsRequest` type: the engine-aware quiesce and MM2 +drain run in the hub, not in the engine-agnostic `dr-controlplane`. + +## Failback + +Failback is not a rewind. When a failed DC returns, it becomes the MM2 target of the +new active. The records it accepted but never mirrored before the failover are a forked +tail Kafka cannot rewind, and because MM2 only adds and never deletes, a naive +re-mirror leaves those orphan records on top of the new active's data (and they could +resurface if that DC is ever made active again). For correctness: + +- **re-seed the affected topics from the new active** (wipe the returned cluster's copy + and re-mirror from scratch), or +- **accept and document the orphan tail** as bounded loss. + +Once the returned DC is caught up, a drained planned switchover returns the active DC: + +```bash +$ kubectl annotate kafka -n demo kf-dcdr dr.kubedb.com/switchover-to=dc-a +``` + +## Scaling and day-2 operations + +The standard `KafkaOpsRequest` operations (`UpdateVersion`, `HorizontalScaling`, +`VerticalScaling`, `VolumeExpansion`, `Restart`, `Reconfigure`, `ReconfigureTLS`, +`RotateAuth`, `StorageMigration`) apply to a DC-DR cluster. They act on the per-DC +Kafka clusters. Horizontal scaling operates per DC (each Member DC's cluster scales its +own brokers or controllers and handles KRaft membership intra-DC via KIP-853), so a +scaling request targets the data centers rather than a single flat broker set. + +There is no failover ops type: unplanned failover is driven by the Lease, and the +planned switchover is the `dr.kubedb.com/switchover-to` annotation, not an ops request. + +> **Note:** the distributed Kafka substrate and the DC-DR layer are net-new for Kafka. +> Treat the field names and flows in this guide as the intended user experience; +> confirm availability in your release before relying on them in production. + +## Deletion and cleanup + +```bash +$ kubectl delete kafka -n demo kf-dcdr +``` + +Per `deletionPolicy`, the operator removes the per-DC Kafka clusters, the per-DC +`ConnectCluster` objects, the MM2 `Connector` objects, and the cluster-scoped per-DC +`PlacementPolicies` it generated (these carry no owner reference, so the operator +deletes them explicitly). The user-provided base `PlacementPolicy` is left for you to +delete. + +## Limitations + +- **No zero-RPO on an unplanned failover.** MM2 is asynchronous, so an unplanned + active-DC loss loses the un-mirrored tail (bounded by MM2 lag). Use a drained planned + switchover for a zero-record-loss move. +- **No rewind on failback.** A returned old-active cluster's un-mirrored forked tail + cannot be rewound. Re-seed the affected topics or accept the orphan tail as bounded + loss. +- **Two data DCs only.** Active/passive MM2 is inherently two-cluster. Three or more + data DCs (fan-out mirroring, three-way failover) is a separate, larger design. +- **Cross-DC reachability is required.** Kafka advertises in-cluster `.svc` listeners, + so MM2 and the cross-DC bootstrap need flat pod networking (KubeSlice) or external + listeners. diff --git a/docs/guides/kafka/dr/overview/index.md b/docs/guides/kafka/dr/overview/index.md new file mode 100644 index 000000000..d2891f3fb --- /dev/null +++ b/docs/guides/kafka/dr/overview/index.md @@ -0,0 +1,307 @@ +--- +title: DC-DR Overview +menu: + docs_{{ .version }}: + identifier: kf-dr-overview-kafka + name: Overview + parent: kf-dr-kafka + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Cross Data Center Disaster Recovery (DC-DR) for Kafka + +KubeDB can run a single distributed `Kafka` across two data centers (DCs) so a Kafka +workload survives the loss of an entire data center. Exactly one DC is the active +write cluster at any instant. The other DC runs a self-contained standby cluster that +receives an asynchronous mirror of the active cluster's topics. When the active DC is +lost, the single bootstrap endpoint is flipped to the standby, the standby is allowed +to take producer writes, and clients continue against identical topic names. + +This page is the conceptual overview and a quick start. See also: + +- [DC-DR User Guide](/docs/guides/kafka/dr/guide/index.md) for every aspect of running + in DC-DR mode (components, the naming contract, connecting, monitoring, the produce + fence, switchover, failback, day-2 ops). +- [DC-DR Runbook](/docs/guides/kafka/dr/runbook/index.md) for what to do in each + operational scenario. + +> **New to KubeDB?** Please start [here](/docs/README.md). + +## Why Kafka DC-DR is its own camp + +Most KubeDB engines have a single writable primary and a leader-to-leader replication +stream. Postgres promotes a survivor, MongoDB elects a new primary, and the endpoint +follows the writable node. **Kafka has none of that.** + +Kafka is not a single-writer database. It is a partitioned log: each topic-partition +has its own leader among that partition's in-sync replicas (ISR), inside one cluster. +There is no cluster-wide primary, and KRaft's own Raft quorum already handles +controller and ISR leadership intra-cluster. So the single-primary DR pattern (one +writable leader, a leader-to-leader stream, promote the survivor) does not map. For +Kafka: + +- **Cross-DC replication is asynchronous log mirroring, not a leader stream.** Kafka's + built-in MirrorMaker 2 (MM2) copies topics (data plus configs), consumer-group + offsets, and heartbeats from a source cluster to a target cluster. KubeDB expresses + MM2 as ordinary `ConnectCluster` and `Connector` objects. There is no new + replication engine to build. +- **The active DC is a write-endpoint routing decision, not an engine state.** + Producers write to whichever cluster the clients are pointed at. DR is + active/passive: one cluster takes writes, MM2 mirrors them to the standby, and on + failover the clients are redirected. The `dr-controlplane` Lease decides which + cluster is the write target; a local produce fence stops producers writing to a + non-active cluster. +- **There is no rewind and no zero-RPO.** MM2 is asynchronous, so an unplanned + failover loses the un-mirrored tail (bounded by MM2 lag), and a returned old-active + cluster may hold records that were never mirrored. Kafka cannot rewind a log. + Failback reverses the mirror direction and reconciles or accepts the un-mirrored + tail as bounded loss. + +So Kafka DC-DR is two independent Kafka clusters (one per Member DC), each with its +own intra-DC KRaft quorum, joined by asynchronous MM2 mirroring, with the Lease +choosing the write cluster and a produce fence preventing split writes. + +## How it works + +DC-DR for Kafka rests on five rules. + +- **KRaft stays intra-DC.** Each Member DC runs its own self-contained Kafka cluster: + its own KRaft controller quorum, its own brokers, its own per-partition ISR. The + KRaft quorum never crosses the DC boundary, so inter-DC latency or a partition can + never flap controller election or stall ISR. There is no cross-DC Kafka voter. +- **The active cluster is chosen only by the `dr-controlplane` primary-DC Lease.** A + small control plane, backed by a three-site etcd quorum, publishes one Lease per + failover scope. The Lease holder is the active write DC. Exactly one cluster is the + write target at any instant. +- **Cross-DC replication is MM2, active to standby.** Following the "consume from + remote, produce to local" best practice, the `ConnectCluster` that does the + mirroring sits with the target (standby) cluster: its `kafkaRef` points at the + standby Kafka, which also holds Connect's internal config, offset, and status + topics. It runs a `MirrorSourceConnector` (topic data and configs), a + `MirrorCheckpointConnector` (consumer-group offsets, so consumers resume after a + flip), and a `MirrorHeartbeatConnector` (liveness and lag). MM2 uses + `IdentityReplicationPolicy` so topic names are identical on both clusters and a + failover is transparent to clients. Because a `ConnectCluster`'s `kafkaRef` is fixed + and its internal topics live on its local cluster, a single `ConnectCluster` cannot + reverse direction: KubeDB pre-provisions one `ConnectCluster` per DC and enables the + mirror connectors only on the current standby's `ConnectCluster`. A failover swaps + which DC's `ConnectCluster` has the connectors enabled, and never enables both + directions at once. +- **Writability is gated by the Lease and fenced locally, fail closed.** A non-active + cluster must refuse producer writes. The fence, driven by the primary-dc marker, + denies the produce operation on the non-active cluster and fails closed: a cluster + that cannot confirm it holds the Lease denies produce. This local fence, plus the + etcd majority, is the split-brain guarantee. Without it a partitioned old-active + cluster that still sees producers would keep accepting writes that never mirror, + diverging the two logs. +- **One bootstrap endpoint follows the active cluster.** The single user-facing + bootstrap Service resolves to the active cluster's brokers (the per-DC `-pods` + bootstrap, selected by the Lease), so producers and consumers always reach the write + cluster. Because MM2 uses `IdentityReplicationPolicy`, the same topic names exist on + the standby, so after the endpoint flips clients keep working and consumers resume + from the offsets the `MirrorCheckpointConnector` already translated. + +> **Why never both directions at once?** `IdentityReplicationPolicy` keeps topic names +> identical and so loses the topic-rename loop guard the default `{source}.` policy +> relies on. If both mirror directions overlap, the same topic can ping-pong between +> the two clusters. A failover therefore disables the old direction's connectors +> before (or atomically with) enabling the new direction's. + +### Data center roles + +Each DC plays one role, set on the `PlacementPolicy` `distributionRule.role`: + +| Role | Holds Kafka | Purpose | +| --- | --- | --- | +| **Member** | yes | A self-contained Kafka cluster with its own KRaft quorum. One Member is the active write cluster; the other is the MM2 mirror target while standby. | +| **Arbiter** | no | The arbiter DC. Holds only the `dr-controlplane` etcd member and never Kafka, because Kafka has no cross-DC voter. Supplies the tie-break etcd vote. | + +## The single-CR, single-endpoint model + +The user creates **one** distributed `Kafka` object (with `spec.distributed` and a +`PlacementPolicy` carrying `distributionRules` and a `failoverPolicy`) and gets **one** +bootstrap endpoint. The operator expands the CR across three CRD kinds: + +- one **`Kafka`** cluster per Member DC, each with its own intra-DC KRaft quorum; +- one **`ConnectCluster`** per DC (each `kafkaRef` pointing at its local Kafka); +- the three **`Connector`** objects (`MirrorSourceConnector`, + `MirrorCheckpointConnector`, `MirrorHeartbeatConnector`) enabled on the current + standby's `ConnectCluster` for the active-to-standby mirror. + +The single CR's `status.disasterRecovery` carries the whole cross-DC view: the active +DC, each cluster's broker health, the MM2 mirror lag, and the DR phase. + +> **Scope.** This spec targets the even two-data-DC layout (two Member DCs plus an +> Arbiter DC). Active/passive MM2 is inherently two-cluster, so spanning three or more +> data DCs (fan-out mirroring and a three-way failover) is a separate, larger design +> and is out of scope here. + +## Prerequisites + +- A distributed Kafka substrate: an Open Cluster Management (OCM) hub and spoke + clusters, and **flat cross-DC pod networking (KubeSlice) or external listeners**. + Kafka brokers advertise in-cluster `.svc` listeners, so MM2's cross-cluster reach + and the cross-DC bootstrap need routable connectivity between the clusters. Wiring + the advertised listeners for cross-DC reach is part of the DC-DR setup. +- The `dr-controlplane` service and its three-site etcd quorum installed across the + data centers, with a `dr-controlplane` agent in each spoke (DC). The third etcd + member sits in the Arbiter DC. +- The KubeDB Kafka operator started with the DC-DR flags (coordination kubeconfig and + the operator's local DC name). +- One consistent **DC name** per data center, used everywhere: the OCM spoke cluster + name, the agent `--dc-name`, the Lease `holderIdentity`, the marker `activeDC`, the + pod label `open-cluster-management.io/cluster-name`, and the `PlacementPolicy` + `distributionRule.clusterName`. Keep them identical. + +## Deploy a DC-DR Kafka + +### 1. PlacementPolicy + +Assign global pod ordinals to data centers and tag each DC with its role. Here two +Member DCs (`dc-a`, `dc-b`) each hold a three-node Kafka cluster, and `dc-c` is the +Arbiter DC: + +```yaml +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + name: kf-dcdr +spec: + clusterSpreadConstraint: + slice: + projectNamespace: kubeslice-demo + sliceName: demo-slice + failoverPolicy: + mode: TwoDC + trigger: + scope: Global + distributionRules: + - clusterName: dc-a + role: Member + replicaIndices: [0, 1, 2] + - clusterName: dc-b + role: Member + replicaIndices: [3, 4, 5] + - clusterName: dc-c + role: Arbiter + replicaIndices: [] +``` + +- A data-bearing **Member** rule carries `replicaIndices` mapping its ordinals to a + self-contained Kafka cluster. The **Arbiter** DC carries an empty `replicaIndices` + and holds no Kafka, only the `dr-controlplane` etcd member. +- `failoverPolicy.mode: TwoDC` expects two Member DCs plus the Arbiter DC. +- `failoverPolicy.trigger.scope: Global` makes this one cluster-wide failover scope. + +### 2. Kafka + +Reference the `PlacementPolicy` and opt the Kafka into DC-DR expansion: + +```yaml +apiVersion: kubedb.com/v1 +kind: Kafka +metadata: + name: kf-dcdr + namespace: demo +spec: + version: 4.0.0 + distributed: true + replicas: 6 + storageType: Durable + podTemplate: + spec: + podPlacementPolicy: + name: kf-dcdr + storage: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +`spec.replicas: 6` is the total broker count across both Member DCs. The +`PlacementPolicy` `replicaIndices` split it into a three-node cluster in `dc-a` +(ordinals 0, 1, 2) and a three-node cluster in `dc-b` (ordinals 3, 4, 5). The operator +expands this into one self-contained Kafka cluster in `dc-a` and one in `dc-b`, a +`ConnectCluster` in each DC (each `kafkaRef` pointing at its local Kafka), and the three +MM2 `Connector` objects on the standby DC's `ConnectCluster` mirroring the active +cluster into the standby. + +## Observe the DC-DR state + +The single `Kafka` object's `status.disasterRecovery` carries the whole cross-DC view: + +```bash +$ kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery}' | jq +``` + +```json +{ + "activeDC": "dc-a", + "phase": "Steady", + "lastTransitionTime": "2026-06-30T10:00:00Z", + "dataCenters": [ + { "clusterName": "dc-a", "role": "Member", "writable": true, "brokersReady": 3, "mirrorLagMillis": 0, "healthy": true }, + { "clusterName": "dc-b", "role": "Member", "writable": false, "brokersReady": 3, "mirrorLagMillis": 850, "healthy": true }, + { "clusterName": "dc-c", "role": "Arbiter", "writable": false, "brokersReady": 0, "mirrorLagMillis": 0, "healthy": true } + ] +} +``` + +- `activeDC` is the DC that holds the Lease and takes producer writes. +- `phase` is `Steady`, `FailingOver`, `FailingBack`, or `Degraded`. +- Each `dataCenters` entry reports the DC role, whether it is the writable cluster, how + many brokers are ready, its MM2 mirror lag in milliseconds (the standby's replication + latency behind the active), and its health. + +## Unplanned failover + +When the active DC is lost, the standby is already a near-current MM2 mirror. The +orchestrator observes the Lease move to the standby, flips the bootstrap endpoint to +the standby's brokers, opens the standby's produce fence, and reverses the MM2 +direction (disabling the connectors on the old active's `ConnectCluster` if it is +reachable, and enabling them on the survivor's for when the old DC returns). +`status.disasterRecovery.phase` moves to `FailingOver` and back to `Steady`. + +The RPO is the un-mirrored MM2 tail: records the active cluster accepted but had not +yet mirrored when it died are lost. There is no rewind. + +## Planned switchover (drained, zero record loss) + +To move the active DC on purpose without losing records, annotate the Kafka with the +target DC: + +```bash +$ kubectl annotate kafka -n demo kf-dcdr dr.kubedb.com/switchover-to=dc-b +``` + +The orchestrator quiesces producers by closing the active cluster's produce fence, +waits for MM2 to drain to near-zero lag (so the target has every record), then flips +the bootstrap endpoint, opens the target's fence, and reverses the mirror direction. +Because MM2 has fully drained before the flip, no committed record is lost. The Lease +then follows to `dc-b`. + +## Failback + +Failback is not a rewind. A returned old-active cluster becomes the MM2 target of the +new active. Records it accepted but never mirrored before the failover are a forked +tail Kafka cannot rewind, and because MM2 only adds and never deletes, a naive +re-mirror leaves those orphan records on top of the new active's data. For +correctness, re-seed the affected topics from the new active (wipe and re-mirror) or +accept and document the orphan tail as bounded loss. Once the returned DC is caught up, +a drained planned switchover returns the active DC. + +## Cleanup + +```bash +$ kubectl delete kafka -n demo kf-dcdr +$ kubectl delete placementpolicy kf-dcdr +``` + +Deleting the `Kafka` removes the per-DC Kafka clusters, the per-DC `ConnectCluster` +objects, the MM2 `Connector` objects, and the generated cluster-scoped per-DC +`PlacementPolicies` (which carry no owner reference, so the operator deletes them +explicitly). The user-provided base `PlacementPolicy` is left for you to delete. diff --git a/docs/guides/kafka/dr/runbook/index.md b/docs/guides/kafka/dr/runbook/index.md new file mode 100644 index 000000000..76d9e859a --- /dev/null +++ b/docs/guides/kafka/dr/runbook/index.md @@ -0,0 +1,342 @@ +--- +title: DC-DR Runbook +menu: + docs_{{ .version }}: + identifier: kf-dr-runbook-kafka + name: Runbook + parent: kf-dr-kafka + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Kafka DC-DR Runbook + +Scenario-by-scenario procedures for operating a Kafka workload in cross data center +disaster recovery (DC-DR) mode. Each scenario lists the **symptoms**, what KubeDB does +**automatically**, how to **verify**, and the **action** to take. + +Read the [User Guide](/docs/guides/kafka/dr/guide/index.md) for the concepts and +commands referenced here. Throughout, `` is the coordination control plane +kubeconfig, and `kf-dcdr`/`demo` are the example database and namespace. + +> **New to KubeDB?** Please start [here](/docs/README.md). + +## Quick reference + +```bash +# Active DC, phase, and per-DC view: +kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery}' | jq + +# Lease holder (the DC the coordination plane makes the active write cluster): +kubectl --kubeconfig -n dc-failover get lease primary-dc -o jsonpath='{.spec.holderIdentity}' + +# Per-DC brokers, roles, and DCs: +kubectl get pods -n demo -l app.kubernetes.io/instance=kf-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name + +# MM2 connector status on the standby's ConnectCluster: +kubectl get connector -n demo -l app.kubernetes.io/instance=kf-dcdr +``` + +Golden rules: + +- **The Lease decides the active write cluster.** Exactly one DC is `writable: true` + in `status.disasterRecovery` at any instant. +- **The produce fence fails closed.** A cluster that cannot confirm it holds the Lease + denies producer writes, so a partitioned old-active cluster stops taking writes on + its own. +- **MM2 is asynchronous.** An unplanned failover loses the un-mirrored tail (bounded by + MM2 lag). Only a drained planned switchover is zero record loss. +- **Never enable both MM2 directions at once** and never fence the MM2 connector + principal or `super.users`. + +--- + +## 1. Intra-DC broker loss (single broker in a DC fails) + +**Symptoms:** one broker pod in a DC is down; some partitions briefly re-elect a +leader within that cluster. + +**Automatic:** the loss is handled entirely inside that DC's Kafka cluster. KRaft +re-elects partition leaders among the surviving in-sync replicas (ISR), and the pod +reschedules. There is **no cross-DC effect**: the active DC stays active, the standby +stays a mirror, and the Lease does not move. + +**Verify:** + +```bash +kubectl get pods -n demo -l app.kubernetes.io/instance=kf-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name +kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' # unchanged +``` + +**Action:** none required. Ensure the failed broker rejoins its cluster's ISR. Provision +user topics with a replication factor and `min.insync.replicas` that tolerate one broker +loss inside a DC. + +--- + +## 2. Full active-DC loss (zone/cluster failure) + +**Symptoms:** the active DC's brokers are gone/unreachable; producers to the bootstrap +endpoint fail briefly. + +**Automatic:** the standby is already a near-current MM2 mirror. The `dr-controlplane` +Lease moves to the standby, and the orchestrator flips the bootstrap endpoint to the +standby's brokers, opens the standby's produce fence, and reverses the MM2 direction +(enabling the survivor-to-old-active connectors for when the old DC returns). `phase` +moves `FailingOver` to `Steady` and the survivor becomes `writable: true`. + +**Verify:** + +```bash +kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' # the survivor +kubectl get pods -n demo -l app.kubernetes.io/instance=kf-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name +``` + +**Action:** none required for availability. Note the RPO: records the old active +accepted but had not yet mirrored are lost (bounded by the MM2 lag at the moment of +loss). There is no rewind. When the failed DC returns, see scenario 6 (failback). + +--- + +## 3. Clean DC-vs-DR partition (both DCs up, cannot reach each other) + +**Symptoms:** the two data DCs are up but the network between them is cut. MM2 lag on +the standby climbs; the old active may still see local producers. + +**Automatic:** the produce fence is the split-brain guard. The old active loses its +Lease renewal across the partition and its fence **fails closed**, so it stops +accepting producer writes on its own **before** the hub reacts. The etcd majority (two +sites plus the Arbiter DC) keeps or grants the Lease to one side only, so exactly one +cluster is ever writable and the two logs cannot diverge. + +**Verify there is exactly one writable DC:** + +```bash +kubectl get kafka -n demo kf-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}={.writable} {end}' +``` + +**Action:** heal the network. Once the partition clears, the non-active cluster resumes +as the MM2 target and catches up. If both sides took writes before the fence closed +(should not happen with a fail-closed fence), treat the non-active side's forked tail +as scenario 6 (re-seed the affected topics). + +--- + +## 4. Planned switchover (maintenance on the active DC) + +**Action:** + +```bash +kubectl annotate kafka -n demo kf-dcdr dr.kubedb.com/switchover-to=dc-b +``` + +**Automatic:** the hub gates on the target's health and MM2 lag budget, then quiesces +producers by closing the active cluster's produce fence, waits for MM2 to drain to +near-zero lag (so the target holds every record), flips the bootstrap endpoint to the +target, opens the target's fence, reverses the mirror direction, and moves the Lease. +Because MM2 fully drained before the flip, **zero committed records are lost**. + +**Verify:** + +```bash +kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' # dc-b +kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery.phase}' # Steady +``` + +**If it does not complete:** see scenario 8 (MM2 lag high / switchover stuck). + +--- + +## 5. Consumers not resuming after a flip + +**Symptoms:** after a failover or switchover, a consumer group re-reads from the +beginning or skips records on the new active cluster. + +**Cause:** consumer offsets were not translated into the new active cluster's offset +space, or the group reconnected before the `MirrorCheckpointConnector` emitted a +checkpoint. + +**Diagnose:** + +```bash +# Is the checkpoint connector running on the (now active) side and syncing group offsets? +kubectl get connector -n demo -l app.kubernetes.io/instance=kf-dcdr +# Confirm sync.group.offsets.enabled=true in the checkpoint connector config secret. +``` + +**Action:** ensure the `MirrorCheckpointConnector` runs with +`sync.group.offsets.enabled=true` so it translates group offsets. If a group flipped +before a checkpoint was available, reset it to the translated offset. Do not disable +the checkpoint connector: it is what lets consumers resume across a flip. + +--- + +## 6. Failback (return a recovered DC to active) + +**Symptoms:** the previously lost DC is back but holds a forked tail (records it +accepted before the failover that were never mirrored). + +**Automatic:** the returned cluster becomes the MM2 target of the new active. But MM2 +only adds and never deletes, so a naive re-mirror leaves the orphan forked tail on top +of the new active's data. + +**Action:** + +1. **Re-seed** the affected topics from the new active: wipe the returned cluster's + copy and re-mirror from scratch, so the forked tail is removed. Or **accept and + document** the orphan tail as bounded loss. +2. Once the returned DC is caught up (low MM2 lag), perform a **drained planned + switchover** back to it: + + ```bash + kubectl annotate kafka -n demo kf-dcdr dr.kubedb.com/switchover-to=dc-a + ``` + +There is no rewind; Kafka cannot roll back a log, so the re-seed is what restores +consistency. + +--- + +## 7. A standby DC is lost + +**Symptoms:** the non-active DC's brokers are gone; that DC shows `healthy: false` and +MM2 lag is unavailable. + +**Impact:** none on writes. The active DC keeps taking producer writes; you lose the +DR copy until the standby returns. + +**Verify the active DC is still writable:** + +```bash +kubectl get kafka -n demo kf-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[?(@.writable==true)]}{.clusterName}{end}' +``` + +**Action:** recover the standby cluster's brokers. When it returns, MM2 resumes +mirroring from the active and the standby catches up. You are running without DR +protection until then. + +--- + +## 8. MM2 lag high (mirror falling behind) + +**Symptoms:** `mirrorLagMillis` on the standby climbs; a planned switchover stays in +`FailingOver` because MM2 has not drained. + +**Diagnose:** + +```bash +# Per-DC MM2 lag and health: +kubectl get kafka -n demo kf-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName} lag={.mirrorLagMillis}ms healthy={.healthy}{"\n"}{end}' +# MM2 connector status: +kubectl get connector -n demo -l app.kubernetes.io/instance=kf-dcdr +``` + +**Causes & action:** + +- **Cross-DC network bottleneck or producer burst:** relieve the bottleneck (network, + active-cluster write load) so MM2 drains. A planned switchover intentionally waits for + the lag to reach near-zero after quiescing producers. +- **A mirror connector is failed/paused:** check the `Connector` status and its config + secret; restart the failed connector on the standby's `ConnectCluster`. +- **Abort a stuck switchover:** remove the annotation to cancel: + `kubectl annotate kafka -n demo kf-dcdr dr.kubedb.com/switchover-to-`. The active DC's + produce fence reopens and it stays active. + +--- + +## 9. Arbiter DC lost + +**Symptoms:** the Arbiter DC is gone; its `dr-controlplane` etcd member is unreachable. + +**Impact:** none on writes. The two data DCs plus the lost arbiter leave the etcd +quorum with two of three members, still a majority, so the Lease can still be +renewed and the active DC keeps taking writes. You lose the tie-break, so a subsequent +**second** failure can no longer keep an etcd majority. + +**Verify the cluster is still writable:** + +```bash +kubectl get kafka -n demo kf-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[?(@.writable==true)]}{.clusterName}{end}' +``` + +**Action:** restore the Arbiter DC's etcd member to regain single-fault tolerance. The +Arbiter DC never runs Kafka, so no broker recovery is involved. + +--- + +## 10. Coordination plane (dr-controlplane / etcd) unavailable + +**Symptoms:** the Lease cannot be read or renewed across the spokes. + +**Automatic:** the current active cluster keeps taking writes as long as its fence can +still confirm the Lease it last held; but if the fence cannot confirm the Lease it +**fails closed** and denies produce, so a total coordination-plane outage can make the +active cluster go read-only. What you always lose is **failover and planned +switchover**: the hub cannot move the active DC until the etcd quorum returns. + +**Verify:** + +```bash +kubectl --kubeconfig -n dc-failover get lease primary-dc # error / stale +kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery.phase}' # may be Degraded +``` + +**Action:** restore the `dr-controlplane` etcd quorum (its third member lives in the +Arbiter DC). Once the Lease is renewable, the fence reopens on the active cluster and +failover/switchover resume. + +--- + +## 11. Which DC is active? + +**Question:** confirm which cluster is taking writes right now. + +```bash +# The DR status view (authoritative for what the hub applied): +kubectl get kafka -n demo kf-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' +# The Lease holder (what the coordination plane intends): +kubectl --kubeconfig -n dc-failover get lease primary-dc -o jsonpath='{.spec.holderIdentity}' +# Where the bootstrap endpoint resolves and which DC is writable: +kubectl get kafka -n demo kf-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}={.writable} {end}' +``` + +In steady state the `activeDC`, the Lease `holderIdentity`, and the single +`writable: true` DC all name the same data center. A brief mismatch during +`FailingOver` is expected; it should converge back to `Steady`. + +--- + +## 12. Suspected split writes (two clusters taking writes) + +This should be impossible: the etcd majority grants the Lease to one DC only, and the +produce fence on any non-Lease-holder fails closed. If `status.disasterRecovery` ever +shows two `writable: true` DCs, or producers succeed against both bootstraps: + +**Diagnose immediately:** + +```bash +kubectl get kafka -n demo kf-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}={.writable} {end}' +kubectl --kubeconfig -n dc-failover get lease primary-dc -o yaml +kubectl get connector -n demo -l app.kubernetes.io/instance=kf-dcdr # both directions must not be enabled +``` + +**Action:** confirm the produce fence is engaged on the non-Lease-holder (ACL revoked +for client principals, or the client listener gated) and that only one MM2 direction's +connectors are enabled. Never enable both directions at once: with +`IdentityReplicationPolicy` there is no topic-rename loop guard, so overlapping mirrors +ping-pong a topic between clusters. Disable the wrong-direction connectors, restore the +fence on the non-active cluster, then treat any forked tail as scenario 6 (re-seed). + +--- + +## Escalation checklist + +When unsure, collect: + +```bash +kubectl get kafka -n demo kf-dcdr -o yaml +kubectl --kubeconfig -n dc-failover get lease -o yaml +kubectl get pods -n demo -l app.kubernetes.io/instance=kf-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name -o wide +kubectl get connectcluster,connector -n demo -l app.kubernetes.io/instance=kf-dcdr -o yaml +```