From 6d04f138c2266f3aaee0c8fd84189dd9751afcc9 Mon Sep 17 00:00:00 2001 From: Tamal Saha Date: Wed, 1 Jul 2026 03:09:36 +0600 Subject: [PATCH] Add MySQL cross-DC disaster recovery (DC-DR) guide Document running a single distributed MySQL across data centers for cross data center disaster recovery. Mirrors the Postgres DR section, adapted to MySQL Group Replication: each Member data center is its own self contained GR cluster, the dr-controlplane primary DC Lease picks the one writable data center, and every standby data center streams a named async replication channel (CHANGE REPLICATION SOURCE ... SOURCE_AUTO_POSITION = 1) from the active data center's primary while GR distributes intra-DC. Adds docs/guides/mysql/dr: - overview: architecture, the GR-never-crosses-DC rule, data center roles, 2-DC vs 3-DC topologies, deploy, status.disasterRecovery, failover, switchover. - guide: components, the DC-name contract, fence (super_read_only, re-asserted after every GR election), GTID and seconds lag, timing invariant, per-DC scaling, DC-aware health, day-2 ops. - runbook: scenario by scenario procedures (active DC loss, partition, planned switchover and failback, stuck switchover, lag, fence tripped, split-brain). Signed-off-by: Tamal Saha --- docs/guides/mysql/dr/_index.md | 10 + docs/guides/mysql/dr/guide/index.md | 383 ++++++++++++++++++++++++ docs/guides/mysql/dr/overview/index.md | 390 +++++++++++++++++++++++++ docs/guides/mysql/dr/runbook/index.md | 360 +++++++++++++++++++++++ 4 files changed, 1143 insertions(+) create mode 100644 docs/guides/mysql/dr/_index.md create mode 100644 docs/guides/mysql/dr/guide/index.md create mode 100644 docs/guides/mysql/dr/overview/index.md create mode 100644 docs/guides/mysql/dr/runbook/index.md diff --git a/docs/guides/mysql/dr/_index.md b/docs/guides/mysql/dr/_index.md new file mode 100644 index 000000000..5a50307a8 --- /dev/null +++ b/docs/guides/mysql/dr/_index.md @@ -0,0 +1,10 @@ +--- +title: Disaster Recovery +menu: + docs_{{ .version }}: + identifier: guides-mysql-dr + name: DR + parent: guides-mysql + weight: 36 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/mysql/dr/guide/index.md b/docs/guides/mysql/dr/guide/index.md new file mode 100644 index 000000000..ce898c27d --- /dev/null +++ b/docs/guides/mysql/dr/guide/index.md @@ -0,0 +1,383 @@ +--- +title: DC-DR User Guide +menu: + docs_{{ .version }}: + identifier: guides-mysql-dr-guide + name: User Guide + parent: guides-mysql-dr + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Running MySQL in DC-DR Mode: User Guide + +This guide covers every aspect of operating a distributed MySQL in cross data center +disaster recovery (DC-DR) mode: the components, the naming contract, deployment, +connecting, monitoring, replication and lag, timing and tuning, quorum and roles, +switchover and failback, scaling, day-2 operations, backup, and deletion. + +Read the [DC-DR Overview](/docs/guides/mysql/dr/overview/index.md) first for the +architecture, and the [DC-DR Runbook](/docs/guides/mysql/dr/runbook/index.md) for +scenario-by-scenario procedures. + +> **New to KubeDB?** Please start [here](/docs/README.md). + +## Components and where they run + +| Component | Runs in | Responsibility | +| --- | --- | --- | +| **`dr-controlplane`** + 3-site etcd quorum | across the data centers (an OCM control plane) | Publishes one `coordination.k8s.io` **Lease** per failover scope. The Lease holder is the active (writable) DC. This is the single cross-DC failover authority. | +| **`dr-controlplane` agent** | each spoke (DC) | Contends for the primary-DC Lease on behalf of its DC and projects the Lease decision into the local spoke as a marker `ConfigMap`. | +| **KubeDB MySQL operator (hub)** | the OCM hub | Expands the `MySQL` CR into per-DC GR clusters, watches the Lease, drives failover/switchover, and writes `status.disasterRecovery`. | +| **`mysql-coordinator`** | every MySQL pod | Manages GR membership and the `kubedb.com/role` label, reads the local marker, and fences its GR primary `super_read_only` when its DC is not active; runs the cross-DC channel on a standby DC's primary. | +| **KubeSlice** | each spoke | Provides the cross-DC pod network so a standby DC's GR primary can stream from the active DC's primary endpoint. | + +The marker `ConfigMap` is the contract between the agent (producer) and the coordinator +(consumer): + +``` +ConfigMap primary-dc (namespace: dc-failover, on each spoke) + data.activeDC = the DC the quorum currently trusts as primary + data.renewTime = RFC3339, the observed primary-DC Lease renewTime +``` + +The coordinator trusts the marker for 30s (the fence TTL); absent, stale, unparseable, or +naming another DC all mean *not active* and the GR primary stays `super_read_only`. This is +the fail-closed fence. + +## The DC-name contract + +One string identifies a data center everywhere. **Keep these identical:** + +- the OCM spoke cluster name +- the agent `--dc-name` +- the primary-DC Lease `holderIdentity` +- the marker `data.activeDC` +- the pod label `open-cluster-management.io/cluster-name` +- the `PlacementPolicy` `distributionRule.clusterName` + +## Operator configuration + +Start the MySQL operator with: + +``` +--dc-dr-enabled +--dc-dr-coord-kubeconfig= +--dc-dr-local-dc= +``` + +The per-DC pod coordinators automatically receive `DC_DR_ENABLED`, `DC_NAME`, +`DC_DR_NAMESPACE` (default `dc-failover`), `DC_DR_MARKER` (default `primary-dc`), and +`DC_DR_SOURCE_HOST` (the active DC's primary endpoint the standby channels from) through +their PetSet template, so the fence and the cross-DC channel work without extra wiring. + +## Deploying + +### PlacementPolicy + +Map the global pod ordinals to data centers and tag each DC with its role: + +```yaml +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + name: my-dcdr +spec: + clusterSpreadConstraint: + slice: + projectNamespace: kubeslice-demo + sliceName: demo-slice + failoverPolicy: + trigger: + scope: Global # one cluster-wide failover scope (or Group + a group name) + mode: TwoDC # TwoDC: 2 Members + a tie-breaker; ThreeDC: 3 Members + distributionRules: + - clusterName: dc-east + role: Member + replicaIndices: [0, 1, 2] + - clusterName: dc-west + role: Member + replicaIndices: [3, 4, 5] + - clusterName: dc-arbiter + role: Arbiter +``` + +- A data-bearing **Member** rule carries `replicaIndices`; the **Arbiter** witness DC + (vote only, no MySQL) carries none. Group Replication has no data-less voter member, so a + MySQL witness DC is always `role: Arbiter` (the petset `Witness` role, a data-bearing + witness, is for engines like MongoDB and is not used by MySQL). +- `mode: TwoDC` expects exactly two Member DCs plus the Arbiter witness DC; `ThreeDC` + expects at least three Member DCs. +- Give each Member DC an **odd** local node count so its GR group keeps a clean majority. + +### MySQL + +```yaml +apiVersion: kubedb.com/v1 +kind: MySQL +metadata: + name: my-dcdr + namespace: demo + annotations: + dr.kubedb.com/enabled: "true" # opt into per-DC DC-DR expansion + # dr.kubedb.com/failover-group: payments # optional: a Group failover scope +spec: + version: "8.4.8" + replicas: 6 + distributed: true + topology: + mode: GroupReplication + storageType: Durable + podTemplate: + spec: + podPlacementPolicy: + name: my-dcdr + storage: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +### What the operator creates + +Per data-bearing DC ``: + +- a per-DC `PetSet` `-` (e.g. `my-dcdr-dc-east`) with its own intra-DC GR group + (its own `group_replication_group_name` UUID); +- a DC-local headless governing `Service`, exported over KubeSlice, so the DC's pods + discover only each other; +- a cluster-scoped per-DC `PlacementPolicy` `-` pinning that group to the DC; +- the cross-DC async channel on each standby DC's GR primary. + +The witness DC (`role: Arbiter`) runs no MySQL pods. All per-DC pods carry the offshoot +selectors plus the `open-cluster-management.io/cluster-name` label, so the global +primary/standby Services and the single `AppBinding` keep working. + +## Connecting + +A DC-DR MySQL exposes the same single endpoint as any KubeDB MySQL: + +- the **primary Service** `` resolves to the active DC's writable GR primary (only that + primary is labeled `kubedb.com/role: primary`); +- the **standby Service** `-standby` resolves to the read-only members; +- one **`AppBinding`** `` for applications and KubeDB integrations. + +Because only the active DC's primary carries the `primary` label, the endpoint follows +failover automatically — applications keep using `` and reconnect after a failover, +landing on the new active DC. + +## Monitoring and observability + +### status.disasterRecovery + +The single CR carries the whole cross-DC view: + +```bash +$ kubectl get my -n demo my-dcdr -o jsonpath='{.status.disasterRecovery}' | jq +``` + +| Field | Meaning | +| --- | --- | +| `activeDC` | The DC that holds the Lease and runs the writable GR primary. | +| `phase` | `Steady`, `FailingOver`, `FailingBack`, or `Degraded`. | +| `lastTransitionTime` | When `activeDC` last changed. | +| `dataCenters[].clusterName` | The data center, by its OCM managed cluster name. | +| `dataCenters[].role` | The DC role: `Member` or `Arbiter`. | +| `dataCenters[].primary` | That DC's local GR primary pod. | +| `dataCenters[].writable` | True only for the active DC. | +| `dataCenters[].lagBytes` | The DC's cross-DC GTID gap behind the active primary, in bytes. | +| `dataCenters[].secondsBehindSource` | The DC's `Seconds_Behind_Source` on its cross-DC channel. | +| `dataCenters[].healthy` | Whether the DC's GR group is healthy. | + +### Useful checks + +```bash +# Which DC is active (from the coordination plane): +$ kubectl --kubeconfig -n dc-failover get lease primary-dc \ + -o jsonpath='{.spec.holderIdentity}' + +# The marker each spoke reads (run against a spoke): +$ kubectl -n dc-failover get configmap primary-dc -o yaml + +# Per-DC GR primaries and roles: +$ kubectl get pods -n demo -l app.kubernetes.io/instance=my-dcdr \ + -L kubedb.com/role,open-cluster-management.io/cluster-name + +# On a standby DC's GR primary, the cross-DC channel: +# SHOW REPLICA STATUS FOR CHANNEL 'dcdr'\G (Replica_IO_Running, Replica_SQL_Running, Seconds_Behind_Source) +``` + +## Replication, lag, and RPO + +- Cross-DC replication is an **asynchronous** replication channel on the standby DC's GR + primary; GR then distributes intra-DC, so each standby DC opens exactly one cross-DC + link. On a GR member the channel is a **named** channel (`dcdr`). +- Lag is reported two ways: `lagBytes` (the GTID gap, `gtid_executed` on the active primary + vs the standby) and `secondsBehindSource` (`Seconds_Behind_Source` from + `SHOW REPLICA STATUS`). The DC's coordinator computes both (the hub never opens + cross-cluster SQL). They are the basis for the RPO of an unplanned failover. +- A **planned switchover loses no committed rows** because writes are frozen and the target + catches up by GTID before the handoff. An **unplanned failover** may lose the last + unreplicated transactions (bounded by the standby's lag at the moment the active DC died). + +## Timing and tuning (RTO vs safety) + +DC-DR has one timing invariant that must hold for correctness: + +> **fence TTL + cross-DC clock skew < primary-DC Lease duration** + +The marker `renewTime` tracks the Lease's renewTime. A partitioned active DC self-fences at +`lastRenew + fence TTL`; a survivor can only acquire the expired Lease at +`lastRenew + LeaseDuration`. Keeping the fence TTL inside the Lease duration guarantees the +old active DC goes `super_read_only` **before** any new DC becomes writable — no split-brain +window. + +Default values: + +| Parameter | Where | Default | +| --- | --- | --- | +| Fence TTL | mysql-coordinator | 30s | +| Marker refresh interval | dr-controlplane agent | 5s | +| Primary-DC Lease duration | dr-controlplane agent (`--election-lease-duration`) | 45s | +| Lease renew deadline | dr-controlplane agent (`--election-renew-deadline`) | 30s | +| Lease retry period | dr-controlplane agent (`--election-retry-period`) | 2s | + +The failover **RTO floor** is roughly the Lease duration (the time a survivor waits to +acquire). To lower RTO, lower the Lease duration **and** the fence TTL together, always +preserving `fence TTL + skew < LeaseDuration`. The retry period must stay well under the +fence TTL so the holder restamps `renewTime` and the marker reads fresh in normal +operation. + +## Quorum, roles, and arbiters + +- Each DC's GR group needs its own majority. Give each Member DC an **odd** local node count + so an intra-DC primary failure keeps quorum. Unlike a raft-based engine, KubeDB does not + add a data-less arbiter member inside a GR group, so avoid even local group sizes. +- The witness DC (`role: Arbiter`) holds only the `dr-controlplane` vote, never MySQL data. +- One GR subtlety the fence handles: GR sets its own elected primary `super_read_only = OFF`, + so the coordinator re-asserts the fence on every label loop. The fail-closed split-brain + guarantee depends on that re-assertion winning the race after each intra standby-DC GR + election. + +Separately from a DC's *intra-DC* quorum, the **cross-DC** failover quorum needs a majority +of three voting sites. For how to lay this out across two or three data centers (and why a +third witness site is preferred), see +[Deployment topologies](/docs/guides/mysql/dr/overview/index.md#deployment-topologies-2-dcs-vs-3-dcs). + +## Planned switchover (near-zero-RPO) + +Move the active DC on purpose by annotating the MySQL: + +```bash +$ kubectl annotate my -n demo my-dcdr dr.kubedb.com/switchover-to=dc-west +``` + +The hub then: + +1. checks the target is a known, healthy DC within the lag budget; +2. sets `phase: FailingOver` and quiesces the active DC (holds its GR primary + `super_read_only = ON`), freezing the active `gtid_executed`; +3. waits until the target's channel has applied up to that frozen GTID set; +4. hands the Lease to the target, whose GR primary is promoted; the old DC resumes as a + standby and starts a channel from the new active. + +The annotation is cleared automatically once the target is active. Watch +`status.disasterRecovery` for `phase` returning to `Steady` with the new `activeDC`. + +## Failback + +Failback is just a switchover back to the original DC once it is healthy again: + +```bash +$ kubectl annotate my -n demo my-dcdr dr.kubedb.com/switchover-to=dc-east +``` + +A DC that lost the Lease and rejoins starts a channel from the new active and catches up by +GTID auto-positioning. If it diverged beyond the source's purged GTIDs (its forked tail +from a bounded-loss promotion), it is re-seeded (the MySQL clone plugin, or +`RESET REPLICA ALL` plus re-provision) before it is eligible. + +## Per-DC horizontal scaling + +Each DC has its own GR group, so scale a specific DC with a `MySQLOpsRequest`: + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: MySQLOpsRequest +metadata: + name: my-dcdr-scale-west + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: my-dcdr + horizontalScaling: + dataCenters: + - clusterName: dc-west + replicas: 5 +``` + +- Each entry sets that DC's local GR node count; DCs not listed are unchanged. +- Nodes are added or removed over the DC-local network; new members join the DC's GR group + and recover from its primary. +- The base `PlacementPolicy` is renumbered so the declarative topology matches. +- Keep each Member DC's count **odd** for a clean GR majority. Removing a whole DC is a + topology change, not horizontal scaling. + +## Day-2 operations + +The standard `MySQLOpsRequest` operations apply to every per-DC GR cluster on a DC-DR +cluster; issue them exactly as for a non-distributed MySQL: + +| Operation | DC-DR behavior | +| --- | --- | +| **Vertical scaling** | Patches every per-DC PetSet, restarts per-DC pods. | +| **Volume expansion** (online/offline) | Expands every per-DC data PVC, waits on all per-DC PetSets. | +| **Version update** | Updates every per-DC PetSet. | +| **Storage migration** | Orphan-deletes and waits on every per-DC PetSet. | +| **Reconfigure / Restart / Rotate-Auth / Reconfigure-TLS** | Apply across the per-DC pods. | + +## Backup + +Back up a DC-DR MySQL the same way as any KubeDB MySQL (KubeStash / the MySQL archiver). +Backups run against the writable endpoint, so they read from the active DC; the AppBinding +follows failover, so a scheduled backup continues against the new active DC after a +failover. Point-in-time recovery works as usual. + +## DC-aware health + +On a DC-DR cluster the operator's health check is DC-aware: + +- the **active** DC is expected to have a writable primary (the write-check runs there); +- a **standby** DC is expected to be `super_read_only` with a healthy inbound channel + (`Replica_IO_Running = Yes`, `Replica_SQL_Running = Yes`), so the passive DC does not flap + to `NotReady`. Set `spec.healthChecker.disableWriteCheck: true` semantics apply on standby + DCs; +- the GR `ONLINE`-member count is checked against the **per-DC** group size, not the global + `spec.replicas`. + +## Deletion and cleanup + +```bash +$ kubectl delete my -n demo my-dcdr +``` + +Per `deletionPolicy`, the operator removes the per-DC PetSets, governing Services, and the +cluster-scoped per-DC `PlacementPolicies` it generated (these carry no owner reference, so +the operator deletes them explicitly). The user-provided base `PlacementPolicy` is left for +you to delete. + +## Limitations + +- **Adding or removing a whole data center** is a topology change (a new GR group and a + cross-DC seed), distinct from horizontal scaling, and is performed by editing the + `PlacementPolicy` topology, not a `HorizontalScaling` request. +- Cross-DC replication is asynchronous; an unplanned failover has a non-zero RPO bounded by + the standby lag. Use a **planned switchover** for zero-RPO moves. +- A standby DC that diverges beyond the active's purged GTIDs must be re-seeded (clone) on + failback rather than GTID catching up. +- All correctness depends on the timing invariant above; do not set a fence TTL that meets + or exceeds the Lease duration. diff --git a/docs/guides/mysql/dr/overview/index.md b/docs/guides/mysql/dr/overview/index.md new file mode 100644 index 000000000..5cfc552fb --- /dev/null +++ b/docs/guides/mysql/dr/overview/index.md @@ -0,0 +1,390 @@ +--- +title: DC-DR Overview +menu: + docs_{{ .version }}: + identifier: guides-mysql-dr-overview + name: Overview + parent: guides-mysql-dr + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# Cross Data Center Disaster Recovery (DC-DR) for MySQL + +KubeDB can run a single distributed `MySQL` across multiple data centers so the +database survives the loss of an entire data center (DC). Exactly one DC is writable +at any instant; the others are warm, read-only standbys that stream from it across the +DCs. When the active DC is lost, KubeDB promotes a surviving DC, and the single +connection endpoint follows the new writable DC. + +DC-DR targets `spec.topology.mode: GroupReplication` (and largely `InnoDBCluster`, +which is Group Replication plus a MySQL Router). `SemiSync` mode is a different shape +(a single writable source with semi-sync replicas) and follows the +[Postgres DC-DR](/docs/guides/postgres/dr/overview/index.md) single-primary model +instead. + +This page is the conceptual overview and a quick start. See also: + +- [DC-DR User Guide](/docs/guides/mysql/dr/guide/index.md) — every aspect of running + in DC-DR mode (components, monitoring, timing, scaling, day-2 ops). +- [DC-DR Runbook](/docs/guides/mysql/dr/runbook/index.md) — what to do in each + operational scenario. + +> **New to KubeDB?** Please start [here](/docs/README.md). + +## How it works + +DC-DR is built on one rule: **the MySQL Group Replication (GR) group never stretches +across data centers.** + +- **Each data center is a self-contained GR cluster.** The operator expands the single + `MySQL` CR into one single-primary GR cluster per data-bearing DC, each with its own + `group_replication_group_name` UUID, its own XCom/Paxos quorum, and its own **local** + primary election driven by GR. The GR group never crosses the DC boundary, so cross-DC + latency cannot stall the group and a partition cannot split it. +- **One cross-DC authority decides who is writable.** A small control plane + (`dr-controlplane`), backed by a three-site etcd quorum, publishes one + `coordination.k8s.io` **Lease** per failover scope. The DC that holds the Lease is the + **active** (writable) DC. This is the single cross-DC failover decision. +- **Cross-DC replication is a normal async replication channel.** The standby DC's GR + **primary** runs an asynchronous replication channel + (`CHANGE REPLICATION SOURCE TO ... SOURCE_AUTO_POSITION = 1`) from the active DC's + primary endpoint; GR then distributes those transactions synchronously to the rest of + that DC's group. So a standby DC opens exactly one cross-DC link. This adapts the + existing `RemoteReplica` wiring, but keeps GR and the coordinator running rather than + RemoteReplica's stripped read-only single server. Because GTIDs are UUID-based and each + DC's group has a distinct UUID, transactions flow across the channel without any domain + ID collision; `SOURCE_AUTO_POSITION = 1` handles positioning. +- **Writability is fenced locally and fails closed.** A per-DC `dr-controlplane` agent + projects the Lease holder onto its own spoke cluster as a small marker `ConfigMap`. The + `mysql-coordinator` reads only that local marker: if it cannot confirm its DC holds the + Lease (the DC lost it, or is partitioned from the coordination plane), it holds its GR + primary `super_read_only = ON`. `super_read_only` blocks client writes but **not** the + channel's applier, so a standby keeps applying the cross-DC stream while refusing direct + writes. Because the fence lives in the DC and fails closed, a cut-off old-active DC stops + accepting writes on its own, before the hub even reacts. This local fence plus the etcd + majority (only one DC can hold the Lease) is the split-brain guarantee. +- **The fence is re-asserted after every GR election.** GR clears `super_read_only` on + whichever member it elects primary, so the coordinator re-asserts the fence on every + label loop. An intra standby-DC GR election (which moves the local primary and the + channel) therefore cannot leave a writable leader behind. +- **Only the active DC's GR primary is labeled `primary`.** Each DC's GR elects its own + primary, but a non-active DC's primary is labeled `kubedb.com/role: standby`, so the + single primary `Service` and the `AppBinding` always resolve to the active DC's writable + primary. + +### Data center roles + +Each DC plays one role, set on the `PlacementPolicy` `distributionRule.role`: + +| Role | Holds MySQL data | Primary eligible | Purpose | +| --- | --- | --- | --- | +| **Member** | yes | yes | A full GR cluster; a candidate for the active DC. | +| **Arbiter** | no | no | Vote only — the `dr-controlplane` etcd tie-breaker; runs no MySQL. **This is the role a MySQL witness DC uses.** | + +> Group Replication has no data-less voter member (its quorum is intra-DC), so a MySQL +> witness DC holds only the `dr-controlplane` etcd member and no MySQL. It is declared +> with `role: Arbiter` and empty `replicaIndices`. The petset `Witness` role (a +> data-bearing witness) is for engines like MongoDB and is not used by MySQL. + +A typical layout is two Member DCs plus one vote-only witness DC (`role: Arbiter`): the +three-site etcd quorum lives across all three, but MySQL data lives only in the two +Member DCs. + +## Deployment topologies (2 DCs vs 3 DCs) + +The DR feature needs two things, in different quantities: + +- **MySQL data** lives in the **Member** data centers. You need at least **two** Member + DCs for cross-DC redundancy (one active, one warm standby). +- **The failover decision** is made by the `dr-controlplane` etcd **quorum**. A quorum + makes progress only while a **majority of its three voting sites** is reachable. For + single-fault tolerance *and* split-brain safety, those three votes should sit in **three + independent failure domains**. The third domain can be a tiny vote-only **witness** + (`role: Arbiter`) that holds no MySQL data. + +So "how many data centers" has two answers: how many hold **data** (two or three), and +how many hold a **quorum vote** (always three for automatic, split-brain-free failover). +The `failoverPolicy.mode` selects the data layout: + +### A. Two Member DCs + a witness — `mode: TwoDC` (recommended) + +Three sites; two hold MySQL data, the third is a vote-only witness DC (`role: Arbiter`, +no MySQL): + +```yaml +failoverPolicy: + mode: TwoDC +distributionRules: +- { clusterName: dc-east, role: Member, replicaIndices: [0, 1, 2] } +- { clusterName: dc-west, role: Member, replicaIndices: [3, 4, 5] } +- { clusterName: dc-witness, role: Arbiter } # etcd vote only, no MySQL +``` + +Any single site can be lost: + +- **Lose a Member DC** → the surviving Member plus the witness form a 2/3 majority, so + the survivor acquires the Lease and is promoted automatically; the lost DC, if alive but + partitioned, self-fences read-only. +- **Lose the witness** → the two Members are still a 2/3 majority, so writes continue + uninterrupted. + +Because the witness runs no MySQL, it is small and cheap. **Run it in a third public +cloud or region** — this is the lowest-cost way to get correct, automatic failover, and +it is the recommended topology whenever a third location is available. + +### B. Three Member DCs — `mode: ThreeDC` + +Three sites, all data-bearing and primary-eligible: + +```yaml +failoverPolicy: + mode: ThreeDC +distributionRules: +- { clusterName: dc-east, role: Member, replicaIndices: [0, 1, 2] } +- { clusterName: dc-west, role: Member, replicaIndices: [3, 4, 5] } +- { clusterName: dc-south, role: Member, replicaIndices: [6, 7, 8] } +``` + +More data copies and read capacity, and any DC can become primary. Tolerates the loss of +any single Member DC. The cost is three full GR clusters instead of two — use it when you +want a data copy and primary capability in all three locations. With three or more data +DCs, each standby DC runs its own async channel from the active, and an unplanned failover +promotes one survivor while every other standby re-points its channel at the new active. + +### C. Two sites only — reduced resiliency + +If you genuinely have only two locations, you still need a third quorum vote, so you +**place it inside one of the two DCs** (run the third `dr-controlplane` etcd member there). +There is no separate witness site, so that DC now holds **two of the three votes**: + +- **Lose the other DC** (the one with one vote) → the two-vote DC keeps the majority → + failover/continuity works automatically. +- **Lose the two-vote DC** → the survivor holds only one of three votes, cannot form a + quorum, and therefore cannot safely become writable on its own. **Automatic failover does + not happen**; recovery is a manual, operator-confirmed step, and you must be certain the + failed DC is truly down to avoid split-brain. + +This protects against losing one specific DC, not both symmetrically. Prefer adding a cheap +third witness site (topology A) whenever possible. + +### At a glance + +| Topology | Sites | Data DCs | Tolerates | Automatic failover | +| --- | --- | --- | --- | --- | +| Two Member + Arbiter witness (`TwoDC`) | 3 | 2 | any 1 site | yes | +| Three Member (`ThreeDC`) | 3 | 3 | any 1 site | yes | +| Two sites, co-located quorum | 2 | 2 | only the one-vote DC | only when the one-vote DC is lost | + +## Prerequisites + +- A working **distributed MySQL** setup: Open Cluster Management (OCM) hub and spoke + clusters, KubeSlice connecting the spokes, and a storage class on each spoke. +- The `dr-controlplane` service and its three-site etcd quorum installed across the data + centers, with a `dr-controlplane` agent running in each spoke (DC). +- The KubeDB MySQL operator started with the DC-DR flags: + + ``` + --dc-dr-enabled + --dc-dr-coord-kubeconfig= + --dc-dr-local-dc= + ``` + +- One consistent **DC name** per data center, used everywhere: the OCM spoke cluster name, + the agent `--dc-name`, the Lease `holderIdentity`, the marker `activeDC`, the pod label + `open-cluster-management.io/cluster-name`, and the `PlacementPolicy` + `distributionRule.clusterName`. Keep them identical. + +## Deploy a DC-DR MySQL + +A DC-DR MySQL is a distributed `MySQL` whose `PlacementPolicy` carries a `failoverPolicy` +and per-DC roles. The user creates and edits a **single** `MySQL` object and gets one +`AppBinding` and one connection endpoint; the operator expands it into the per-DC GR +clusters. + +### 1. PlacementPolicy + +Assign the global pod ordinals to data centers and tag each DC with its role. Here two +Member DCs (`dc-east`, `dc-west`) each get three MySQL pods, and `dc-arbiter` is the +tie-breaking witness: + +```yaml +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + name: my-dcdr +spec: + clusterSpreadConstraint: + slice: + projectNamespace: kubeslice-demo + sliceName: demo-slice + failoverPolicy: + trigger: + scope: Global + mode: TwoDC + distributionRules: + - clusterName: dc-east + role: Member + replicaIndices: [0, 1, 2] + - clusterName: dc-west + role: Member + replicaIndices: [3, 4, 5] + - clusterName: dc-arbiter + role: Arbiter +``` + +- A data-bearing **Member** rule carries `replicaIndices`; the **Arbiter** witness DC + (vote only, no MySQL) carries none. +- `failoverPolicy.trigger.scope: Global` makes this one cluster-wide failover scope. Use + `Group` with a group name to put a database in its own scope. +- Give each Member DC an **odd** local node count so its GR group keeps a clean majority + for intra-DC failover. + +### 2. MySQL + +Reference the `PlacementPolicy` and opt the MySQL into DC-DR expansion: + +```yaml +apiVersion: kubedb.com/v1 +kind: MySQL +metadata: + name: my-dcdr + namespace: demo + annotations: + # Opt this distributed MySQL into per-DC DC-DR expansion. + dr.kubedb.com/enabled: "true" +spec: + version: "8.4.8" + replicas: 6 + distributed: true + topology: + mode: GroupReplication + storageType: Durable + podTemplate: + spec: + podPlacementPolicy: + name: my-dcdr + storage: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +The operator then creates, per data-bearing DC: + +- a per-DC `PetSet` named `-` (for example `my-dcdr-dc-east`) with its own intra-DC + GR group and DC-local governing `Service` (exported over KubeSlice); +- the cross-DC async channel on each standby DC's GR primary. + +The witness DC (`role: Arbiter`) runs no MySQL pods. + +## Observe the DC-DR state + +The single `MySQL` object's `status.disasterRecovery` carries the whole cross-DC view: + +```bash +$ kubectl get my -n demo my-dcdr -o jsonpath='{.status.disasterRecovery}' | jq +``` + +```json +{ + "activeDC": "dc-east", + "phase": "Steady", + "lastTransitionTime": "2026-06-30T10:00:00Z", + "dataCenters": [ + { "clusterName": "dc-east", "role": "Member", "primary": "my-dcdr-dc-east-0", "writable": true, "healthy": true }, + { "clusterName": "dc-west", "role": "Member", "primary": "my-dcdr-dc-west-0", "writable": false, "healthy": true, "lagBytes": 4096, "secondsBehindSource": 1 } + ] +} +``` + +- `activeDC` is the DC that currently holds the Lease and runs the writable GR primary. +- `phase` is `Steady`, `FailingOver`, `FailingBack`, or `Degraded`. +- Each `dataCenters` entry reports that DC's GR primary pod, whether it is the writable + primary, its health, and its cross-DC lag as both a GTID gap (`lagBytes`) and + `secondsBehindSource` (from `SHOW REPLICA STATUS`). The in-DC coordinator computes these + and surfaces them; the hub never opens cross-cluster SQL. + +## Unplanned failover + +When the active DC is lost, its agents stop renewing the primary-DC Lease. After the Lease +duration a surviving Member DC's agent acquires it; that DC becomes `activeDC`. The hub +observes the change and clears the survivor's fence: it relabels the survivor's GR primary +`primary`, sets `super_read_only = OFF`, stops the survivor's inbound channel, and repoints +every other standby DC's channel at the new active. The old DC self-fences read-only +locally (before the hub reacts, even under a partition). The primary `Service` and +`AppBinding` then resolve to the new writable DC. + +You do not trigger this; it is automatic. `status.disasterRecovery.phase` moves to +`FailingOver` during the transition and back to `Steady` once the survivor is primary. The +RPO is bounded by the survivor's cross-DC lag at the moment the active DC died (the +un-shipped GTID tail). + +## Planned switchover (near-zero-RPO) + +To move the active DC on purpose (maintenance, rebalancing) without losing committed rows, +annotate the MySQL with the target DC: + +```bash +$ kubectl annotate my -n demo my-dcdr dr.kubedb.com/switchover-to=dc-west +``` + +The switchover is coordinated for near-zero RPO: + +1. The target must be a known, healthy DC within the lag budget. +2. The hub quiesces writes on the active DC (holds its GR primary `super_read_only = ON`), + so the active primary's `gtid_executed` freezes. +3. The hub waits until the target's channel has applied up to the active primary's frozen + `gtid_executed`. +4. The Lease hands off to the target; its GR primary is promoted (relabeled `primary`, + fence cleared) and the active DC resumes as a standby (its GR primary starts a channel + from the new active). The annotation is cleared automatically. + +Because writes are frozen and the target fully catches up by GTID before the handoff, a +planned switchover loses no committed rows. + +## Scale a data center + +Each DC has its own intra-DC GR group, so a single `spec.replicas` cannot describe a +scale. Scale a specific DC with a `MySQLOpsRequest` that lists per-DC targets: + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: MySQLOpsRequest +metadata: + name: my-dcdr-scale + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: my-dcdr + horizontalScaling: + dataCenters: + - clusterName: dc-west + replicas: 5 +``` + +Each entry sets that data center's local GR node count; DCs not listed are unchanged. The +request resizes only `dc-west`'s GR group, then updates the `PlacementPolicy` so the +declarative topology matches. No other DC's group and no cross-DC writability is touched. + +## Day-2 operations + +The standard MySQL `MySQLOpsRequest` operations work on a DC-DR cluster and act on every +per-DC GR cluster: vertical scaling, volume expansion (online and offline), version update, +and storage migration each apply to all per-DC `PetSet`s. You issue them exactly as for a +non-distributed MySQL. + +## Cleanup + +```bash +$ kubectl delete my -n demo my-dcdr +$ kubectl delete placementpolicy my-dcdr +``` + +Deleting the `MySQL` removes the per-DC `PetSet`s, governing `Service`s, and the +cluster-scoped per-DC `PlacementPolicies` the operator generated. The user-provided base +`PlacementPolicy` is left for you to delete. diff --git a/docs/guides/mysql/dr/runbook/index.md b/docs/guides/mysql/dr/runbook/index.md new file mode 100644 index 000000000..fff70799f --- /dev/null +++ b/docs/guides/mysql/dr/runbook/index.md @@ -0,0 +1,360 @@ +--- +title: DC-DR Runbook +menu: + docs_{{ .version }}: + identifier: guides-mysql-dr-runbook + name: Runbook + parent: guides-mysql-dr + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +# MySQL DC-DR Runbook + +Scenario-by-scenario procedures for operating a MySQL cluster in cross data center +disaster recovery (DC-DR) mode. Each scenario lists the **symptoms**, what KubeDB does +**automatically**, how to **verify**, and the **action** to take. + +Read the [User Guide](/docs/guides/mysql/dr/guide/index.md) for the concepts and commands +referenced here. Throughout, `` is the coordination control plane kubeconfig, +`my-dcdr`/`demo` are the example database and namespace. + +## Quick reference + +```bash +# Active DC, phase, and per-DC view: +kubectl get my -n demo my-dcdr -o jsonpath='{.status.disasterRecovery}' | jq + +# Lease holder (the source of truth for "who is active"): +kubectl --kubeconfig -n dc-failover get lease primary-dc -o jsonpath='{.spec.holderIdentity}' + +# Per-DC GR primaries, roles, and DCs: +kubectl get pods -n demo -l app.kubernetes.io/instance=my-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name + +# A spoke's marker (what its coordinators read): +kubectl -n dc-failover get configmap primary-dc -o jsonpath='{.data}' +``` + +Golden rules: + +- **The Lease decides who is writable.** Never clear `super_read_only` on a pod by hand. +- **The fence fails closed.** A DC that cannot confirm it holds the Lease is + `super_read_only` by design — that is correct, not a bug. +- **Exactly one DC is `writable: true`** in `status.disasterRecovery` at any instant. + +--- + +## 1. Active DC lost (zone/cluster failure) + +**Symptoms:** the active DC's pods are gone/unreachable; writes fail briefly. + +**Automatic:** the lost DC's agents stop renewing the Lease. After the Lease duration +(~45s) a surviving Member DC's agent acquires it and becomes `activeDC`. The hub clears the +survivor's fence (relabels its GR primary `primary`, `super_read_only = OFF`, stops its +inbound channel) and repoints every other standby's channel at the new active; the old DC, +if partially alive, self-fences read-only. The primary `Service` and `AppBinding` follow to +the new DC. `phase` moves `FailingOver` → `Steady`. + +**Verify:** + +```bash +kubectl get my -n demo my-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' # the survivor +kubectl get my -n demo my-dcdr -o jsonpath='{.status.disasterRecovery.phase}' # Steady +``` + +**Action:** none required for availability. Note the RPO: transactions not yet replicated +when the DC died (the un-shipped GTID tail) are lost. When the failed DC returns, see +scenario 11 (re-add a DC). + +--- + +## 2. Network partition between data centers + +**Symptoms:** DCs are up but cannot reach each other or the coordination plane. + +**Automatic:** the side that loses the coordination plane stops getting Lease updates; its +marker `renewTime` freezes and, after the 30s fence TTL, its coordinator holds its GR +primary `super_read_only = ON` — **before** the Lease duration lets the other side acquire +(this is the timing invariant). The side that keeps the etcd majority holds/acquires the +Lease and stays (or becomes) writable. There is no split-brain. + +**Verify there is exactly one writable DC:** + +```bash +kubectl get my -n demo my-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}={.writable} {end}' +``` + +**Action:** heal the network. The fenced side rejoins, GTID auto-positions (or re-seeds if +diverged beyond the purged GTIDs), and resumes its channel automatically. If both sides +show `writable: false`, see scenario 6 (coordination plane down). + +--- + +## 3. Planned switchover (maintenance on the active DC) + +**Action:** + +```bash +kubectl annotate my -n demo my-dcdr dr.kubedb.com/switchover-to=dc-west +``` + +**Automatic:** the hub gates on the target's health and lag, quiesces the active DC (holds +its GR primary `super_read_only = ON`), waits until the target's channel applies up to the +frozen `gtid_executed`, then hands off. Zero committed rows are lost. The annotation is +cleared on completion. + +**Verify:** + +```bash +kubectl get my -n demo my-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' # dc-west +kubectl get my -n demo my-dcdr -o jsonpath='{.status.disasterRecovery.phase}' # Steady +``` + +**If it does not complete:** see scenario 8 (switchover stuck). + +--- + +## 4. Planned failback to the original DC + +After the original DC is healthy and caught up: + +```bash +kubectl annotate my -n demo my-dcdr dr.kubedb.com/switchover-to=dc-east +``` + +Same zero-RPO flow as scenario 3. A DC that previously lost the Lease catches up by GTID +(or re-seeds via the clone plugin if it diverged beyond the purged GTIDs) before it is +eligible, so failback is safe even after an unplanned failover. + +--- + +## 5. A standby DC is lost + +**Symptoms:** a non-active DC's pods are gone; that DC shows `healthy: false`. + +**Impact:** none on writes — the active DC is unaffected. You lose that DC's redundancy and +its standby read capacity until it returns. + +**Verify the active DC is still writable:** + +```bash +kubectl get my -n demo my-dcdr -o jsonpath='{.status.disasterRecovery.activeDC}' +# Full per-DC writability: +kubectl get my -n demo my-dcdr -o json | jq -r '.status.disasterRecovery.dataCenters[] | "\(.clusterName)=\(.writable)"' +``` + +**Action:** recover the DC's nodes; the per-DC GR group reschedules and its primary +re-establishes the cross-DC channel from the active automatically. + +--- + +## 6. Coordination plane (dr-controlplane / etcd) unavailable + +**Symptoms:** the Lease cannot be read/renewed; markers go stale across all spokes; every +DC eventually fences `super_read_only`. + +**Automatic:** this is fail-closed — with no trustworthy Lease, **no** DC is allowed to be +writable. The database is read-only globally rather than risk split-brain. + +**Verify:** + +```bash +kubectl --kubeconfig -n dc-failover get lease primary-dc # error / stale renewTime +kubectl get my -n demo my-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName}={.writable} {end}' # all false +``` + +**Action:** restore the `dr-controlplane` etcd quorum. Once the Lease is renewable, the +holder's marker refreshes, its coordinator clears `super_read_only`, and writes resume. Do +**not** clear `super_read_only` on a pod to work around this. + +--- + +## 7. Failover not promoting a survivor + +**Symptoms:** the active DC is gone but `activeDC` does not move, or no writable DC appears. + +**Diagnose:** + +```bash +# Did the Lease move? +kubectl --kubeconfig -n dc-failover get lease primary-dc -o jsonpath='{.spec.holderIdentity}' +# Are there ONLINE GR members in the survivor DC? +kubectl get pods -n demo -l app.kubernetes.io/instance=my-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name +``` + +**Common causes & action:** + +- **Lease did not move** — only Member DCs are eligible; confirm the survivor is a `Member` + in the `PlacementPolicy` and its agent can reach the coordination plane. +- **No quorum in the survivor** — the survivor DC's GR group lost its own intra-DC majority + (for example an even local size that lost a node); recover its pods so GR re-forms. +- **Fence not clearing** — confirm the marker on the survivor's spoke names the survivor and + has a fresh `renewTime`; the coordinator clears `super_read_only` only once it does. + +--- + +## 8. Planned switchover stuck (target not catching up) + +**Symptoms:** after annotating `switchover-to`, `phase` stays `FailingOver` and the Lease +does not hand off. + +**Diagnose:** + +```bash +# Target lag and health: +kubectl get my -n demo my-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName} lagBytes={.lagBytes} secs={.secondsBehindSource} healthy={.healthy}{"\n"}{end}' +``` + +**Causes & action:** + +- **Target lag not converging** — the active DC must be quiesced for the target to reach the + frozen GTID set. Confirm the active DC's GR primary is `super_read_only` (its coordinator + honored the quiesce). +- **Target unhealthy / no lag report** — the switchover refuses a target with no lag yet; + ensure the target DC's GR primary is up and its channel is running + (`Replica_IO_Running`/`Replica_SQL_Running = Yes`). +- **Target legitimately too far behind** — let it drain, or relieve the cross-DC bottleneck + (scenario 9) before retrying. +- **Abort** — remove the annotation to cancel: + `kubectl annotate my -n demo my-dcdr dr.kubedb.com/switchover-to-`. + +--- + +## 9. Lag growing on a standby DC + +**Symptoms:** a DC's `lagBytes` / `secondsBehindSource` climbs steadily. + +**Diagnose:** cross-DC network throughput/latency, write volume on the active primary, and +the channel on the standby DC's GR primary. + +```bash +# On the standby DC's GR primary: +# SHOW REPLICA STATUS FOR CHANNEL 'dcdr'\G (Seconds_Behind_Source, Replica_SQL_Running) +# On the active primary, the cross-DC client: +# SELECT * FROM performance_schema.replication_connection_status\G +``` + +**Action:** relieve the bottleneck (network, primary load). High lag widens the RPO of an +unplanned failover and can block a planned switchover until it drains. Ensure the active +DC's binlog retention (`binlog_expire_logs_seconds`) outlasts the lag, or a slow DR DC's +channel can break when needed binlogs are purged. + +--- + +## 10. A DC is unexpectedly read-only (fence tripped) + +**Symptoms:** a DC you expect to be active is `super_read_only`; its GR primary is labeled +`standby`. + +**Diagnose the fence chain:** + +```bash +# Does this spoke's marker name this DC and is renewTime fresh? +kubectl -n dc-failover get configmap primary-dc -o jsonpath='{.data}' +# Is the dr-controlplane agent running in this DC and renewing? +kubectl get pods -n -l app=dr-controlplane-agent +``` + +**Causes & action:** + +- **Marker stale** (`renewTime` old) — the agent cannot reach the coordination plane, or the + projector is failing; restore agent connectivity. +- **Marker names another DC** — this DC simply is not the active one (correct). +- **Clock skew** — large cross-DC clock skew can trip the TTL early; verify NTP. The timing + budget assumes skew is well under (LeaseDuration − fence TTL). + +Never clear `super_read_only` or patch `kubedb.com/role` by hand to force writability — the +next label loop re-asserts the fence, and you risk split-brain. + +--- + +## 11. Re-add / recover a previously lost data center + +After a DC returns from a failure: + +**Automatic:** its per-DC GR group re-forms, and its primary starts a channel from the +active DC and catches up by GTID auto-positioning (a DC that diverged beyond the active's +purged GTIDs is re-seeded via the clone plugin first). Once caught up, the DC's primary +becomes a healthy `super_read_only` standby and its lag appears in status. + +**Verify:** + +```bash +kubectl get my -n demo my-dcdr -o jsonpath='{range .status.disasterRecovery.dataCenters[*]}{.clusterName} healthy={.healthy} lagBytes={.lagBytes}{"\n"}{end}' +``` + +**Action:** to make it active again, perform a planned failback (scenario 4) once its lag is +small. + +--- + +## 12. Scale a data center up or down + +```bash +# Scale dc-west to 5 nodes: +kubectl apply -f - <<'YAML' +apiVersion: ops.kubedb.com/v1alpha1 +kind: MySQLOpsRequest +metadata: { name: my-dcdr-scale-west, namespace: demo } +spec: + type: HorizontalScaling + databaseRef: { name: my-dcdr } + horizontalScaling: + dataCenters: + - { clusterName: dc-west, replicas: 5 } +YAML +``` + +**Verify:** the per-DC PetSet reaches the new size and only that DC changed. + +```bash +kubectl get petset -n demo -l app.kubernetes.io/instance=my-dcdr +kubectl get mysqlopsrequest -n demo my-dcdr-scale-west -o jsonpath='{.status.phase}' +``` + +**Notes:** keep each Member DC's count **odd** for a clean GR majority; removing a whole DC +is a topology change, not horizontal scaling. + +--- + +## 13. Version upgrade in DC-DR + +Issue a normal `UpdateVersion` `MySQLOpsRequest`; the operator updates every per-DC PetSet. +Plan it during a low-traffic window and confirm each DC returns healthy in +`status.disasterRecovery` before relying on failover again. + +--- + +## 14. Suspected split-brain (two writable DCs) + +This should be impossible by design (etcd majority + fail-closed fence + the timing +invariant + fence re-assertion after every GR election). If `status.disasterRecovery` ever +shows two `writable: true` DCs, or two pods labeled `kubedb.com/role: primary`: + +**Diagnose immediately:** + +```bash +kubectl get pods -n demo -l app.kubernetes.io/instance=my-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name +kubectl --kubeconfig -n dc-failover get lease primary-dc -o jsonpath='{.spec.holderIdentity}' +``` + +**Action:** the Lease holder is the true active DC. The other DC's fence should trip within +the TTL; if it does not, its marker is wrong (check its agent/projector) or the timing +invariant is misconfigured (verify fence TTL < Lease duration). Stop writes to the +non-Lease-holder DC at the application layer until the fence reasserts, then reconcile (the +non-holder GTID catches up or re-seeds and rejoins as a standby). + +--- + +## Escalation checklist + +When unsure, collect: + +```bash +kubectl get my -n demo my-dcdr -o yaml +kubectl --kubeconfig -n dc-failover get lease -o yaml +kubectl -n dc-failover get configmap primary-dc -o yaml # on each spoke +kubectl get pods -n demo -l app.kubernetes.io/instance=my-dcdr -L kubedb.com/role,open-cluster-management.io/cluster-name -o wide +kubectl logs -n demo -c mysql-coordinator +```