Rust #325: qualify native on MySQL InnoDB Cluster behind Router

docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md

@@ -0,0 +1,115 @@
+# Native on MySQL InnoDB Cluster behind MySQL Router (November 1)
+
+A highly available native deployment on MySQL runs native 3.x on a MySQL 8
+InnoDB Cluster behind MySQL Router. This checkpoint qualifies that topology
+on one disposable host with the
+[harness](../../scripts/perf/mysql-innodb-cluster/README.md). It is evidence
+for the topology and its failure handling, not a capacity measurement.
+
+## Conditions
+
+- MySQL 8.4.13 (`mirror.gcr.io/library/mysql@sha256:a0225d1e…`), three
+  members, single primary, default Group Replication settings (MySQL
+  communication stack, `group_replication_member_expel_timeout` 5 s), 1 GiB
+  and a 128 MiB buffer pool each.
+- MySQL Router 8.4.10
+  (`container-registry.oracle.com/mysql/community-router@sha256:d704471c…`),
+  bootstrapped defaults; native uses the classic read-write port 6446.
+- Native from this branch (main at `bbed2bec` plus the two fixes below), with
+  `DW_RUST_MAINTENANCE_LOOP=1`, 512 MiB.
+- Load: published Python SDK 2.12.2, one worker (eight workflow and eight
+  activity slots), workflows of a 1 s timer then two activities.
+
+## Results
+
+| Run | Failure | Workflows completed | Starts refused (503) | Readiness non-200 | Integrity |
+| --- | --- | --- | --- | --- | --- |
+| Baseline | none | 60/60, p50 5.1 s | 0 | none | exact |
+| 2 | primary crash (`docker kill`) | 106 of 107 started; the 107th made ready again at its activity lease expiry | 43 | +4.4 s to +40.7 s | exact |
+| 3 | primary clean shutdown (`docker stop`) | 150/150 | 0 | one sample, ready again by +3.8 s | exact |
+| 4 | primary hung (`docker pause`) | 74/74 | 16 | +0.5 s to +32.7 s | exact |
+
+"Exact" means: every completed run has one `WorkflowCompleted`, one
+`TimerFired` and two `ActivityCompleted` events, no `(run, sequence)` history
+row is duplicated, and no workflow task or activity attempt is left leased.
+
+- **Schema creation through Router** works: native created and verified its
+  schema on a fresh database through port 6446.
+- **Crash recovery time** (run 2, measured from the kill): Group Replication
+  elected the new primary at +23.4 s, Router routed writes to it at +36 s,
+  native's health answered 200 at +37.0 s and readiness at +40.7 s. Native
+  adds 1 to 5 s after Router switches; the rest is election and Router's
+  metadata refresh, which operators can tune.
+- **Honest answers during the window**: with no primary, every request,
+  including starts, answers `503 storage_unavailable`; readiness answers 503
+  and health answers 503 or times out at the probe. Nothing answers 200 while
+  writes cannot commit. Native reconnects without a restart; the scheduler
+  loop logs `runtime_scheduler_storage_unavailable` once and resumes.
+- **In-flight work**: an activity claimed 0.4 s before the crash could not be
+  completed by its worker during the window. At its lease expiry native
+  expired the attempt, failed the activity under its retry policy and made a
+  workflow task ready, so the run resumes; nothing stays leased.
+- **Client-visible errors**: the Python SDK's `start_workflow` and
+  `result()` do not retry 503, so a caller sees `storage_unavailable` for
+  calls made during the window (19 `result()` calls in run 2, whose runs did
+  complete server-side). Callers should retry 503 with backoff.
+- **Throughput is the driver's limit, not the failover's**: at 2 starts per
+  second the single SDK worker saturates (a no-failover control had a 73 s
+  median); run 4 used 1 per second.
+
+A first crash run used Docker's embedded DNS, which drops a killed
+container's name. The surviving members, on the MySQL communication stack,
+then reported each other `UNREACHABLE` intermittently and lost quorum; no
+primary was elected and the old one could not rejoin. With fixed addresses
+and static host entries, as real hosts have, the same crash failed over
+cleanly. Cluster members need names that keep resolving while a member is
+down.
+
+## Defects found and fixed
+
+- **Activity poll without `server_time`** (any topology). Published PHP has
+  answered `POST /api/worker/activity-tasks/poll` with a top-level
+  `server_time` since 2.11.0; native did not. Python SDK 2.12.2 measures a
+  claimed activity's lease against it and refused every activity native
+  delivered (`ActivityLeaseAuthorityLost`, `invalid_activity_claim`), so no
+  activity could run under that SDK. Native now adds `server_time` to every
+  activity poll answer, leased or empty
+  (`activity_poll_answers_carry_the_server_clock`).
+- **Failover error classes**. MySQL errors a primary switch can surface are
+  now classified transient: `backend_unavailable` for 1053, 1290, 1927, 2055,
+  3100 and 4031, `backend_lock_pressure` for 3101
+  (`mysql_cluster_failover_errors_are_transient`). Every database error
+  already answered 503; the class drives the bounded diagnostic and the
+  retryable worker deregistration receipt.
+
+## Settings for operators
+
+- **Connection**: `DB_CONNECTION=mysql`, `DB_HOST=<router>`, `DB_PORT=6446`
+  (Router's classic read-write port, never the read-only 6447 or the
+  read/write-split 6450), `DB_DATABASE`, `DB_USERNAME`, `DB_PASSWORD`, plus
+  `DB_SSLMODE`/`DB_SSLROOTCERT` for TLS to Router.
+- **Pool** (fixed in native): four connections per process, 5 s acquire
+  timeout, connections tested before reuse, `lock_wait_timeout` and
+  `innodb_lock_wait_timeout` of 5 s per session. Size Router's
+  `max_connections` for four per native process plus headroom.
+- **Probes**: readiness gates traffic. While no primary is reachable, health
+  and readiness can take up to about 5 s to answer 503 (the pool's acquire
+  timeout), so give probes a timeout of at least 6 s. Do not restart native on
+  a database outage: health is 503 for the whole failover window and a
+  restart does not shorten it. Expect 35 to 45 s of 503 after a primary crash
+  with default Group Replication and Router settings, and a few seconds after
+  a clean shutdown.
+- **Router**: keep `connection_sharing` off (the default); native sets
+  session variables on connect. Router's metadata `ttl` and Group Replication
+  notifications, and Group Replication's `member_expel_timeout`, set the
+  failover window.
+- **Named locks**: native takes MySQL `GET_LOCK` only while bootstrapping or
+  upgrading its schema and in upgrade takeover, never at runtime. Named locks
+  are local to the member, so a failover during a schema upgrade releases the
+  lock; run upgrades with one native process.
+
+## Not covered
+
+Multi-primary mode, Router read/write split, Router on a separate host or
+several Routers, long-running load, cross-host network partitions, TLS to
+Router, and the Rust and PHP SDKs.

docs/rust-server-port.md

@@ -272,6 +272,18 @@ These are confirmed differences from the published PHP reference and open
 contract decisions found while qualifying native slices. Each one needs a
 contract fixture before its correction.
 
+- **MySQL InnoDB Cluster behind MySQL Router** (November 1, qualified on one
+  host; [checkpoint](rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md)).
+  Native runs on a single-primary InnoDB Cluster through Router's
+  read-write port. Schema creation, the workflow, timer and
+  activity lifecycle, and primary crash, clean shutdown and hang all pass
+  with exact history and no leftover leases; native answers 503 while no
+  primary is reachable and recovers without a restart, 1 to 5 s after Router
+  switches. Two defects found and fixed: the activity poll answer lacked
+  PHP's `server_time` (since 2.11.0), so Python SDK 2.12.2 refused every
+  activity native delivered on any database; and MySQL failover errors
+  (1053, 1290, 1927, 2055, 3100, 3101, 4031) are now classified transient.
+  Multi-primary, read/write split and several Routers are not covered.
 - **History export waits and timeline** (November 1, closed,
   [checkpoint](rust-server-port-checkpoints/2026-11-01-history-export-waits.md)).
   Native's export now serves the run's stored waits with each kind's

rust/src/runtime/store.rs

@@ -266,6 +266,16 @@ fn after(seconds: i64) -> DateTime<Utc> {
     now() + chrono::Duration::seconds(seconds)
 }
 
+/// As PHP's `ActivityTaskController::poll` (since 2.11.0): an activity poll
+/// answer carries `server_time`, the clock its lease expiry is measured
+/// against. Published SDKs refuse a claimed activity without it.
+fn activity_server_time(kind: &str, mut answer: Value) -> Value {
+    if kind == "activity" {
+        answer["server_time"] = json!(super::activity_lease_renewal::micros(now()));
+    }
+    answer
+}
+
 impl<DB: Backend> Store<DB>
 where
     for<'c> &'c mut DB::Connection: Executor<'c, Database = DB>,
@@ -1974,19 +1984,22 @@ where
                 // Compressed after the claim commits; a replayed receipt keeps
                 // the plain events and takes this request's encoding.
                 super::history_pages::compress(&mut task, encoding)?;
-                return Ok(
+                return Ok(activity_server_time(
+                    kind,
                     json!({"task": task, "poll_status": "leased", "server_capabilities": capabilities()}),
-                );
+                ));
             }
             if tokio::time::Instant::now() >= deadline {
-                return self.no_task_answer(&body, poll_request.as_ref(), "empty", encoding).await;
+                return self.no_task_answer(&body, poll_request.as_ref(), "empty", encoding).await
+                    .map(|answer| activity_server_time(kind, answer));
             }
             if self
                 .poll_queue_changed(&body, kind, &mut cross_kind)
                 .await?
                 && tokio::time::Instant::now() < deadline
             {
-                return self.no_task_answer(&body, poll_request.as_ref(), "task_queue_changed", encoding).await;
+                return self.no_task_answer(&body, poll_request.as_ref(), "task_queue_changed", encoding).await
+                    .map(|answer| activity_server_time(kind, answer));
             }
             #[cfg(debug_assertions)]
             if let Some(request) = &poll_request {

rust/src/runtime/worker_incarnations.rs

@@ -23,11 +23,7 @@ pub(super) fn transient_reason(error: &RuntimeError) -> Option<&'static str> {
         ) => Some("backend_unavailable"),
         RuntimeError::Database(sqlx::Error::Database(error)) => {
             if let Some(mysql) = error.try_downcast_ref::<sqlx::mysql::MySqlDatabaseError>() {
-                return match mysql.number() {
-                    1205 | 1213 => Some("backend_lock_pressure"),
-                    2002 | 2003 | 2006 | 2013 => Some("backend_unavailable"),
-                    _ => None,
-                };
+                return mysql_transient_reason(mysql.number());
             }
             let code = error.code()?;
             if matches!(code.as_ref(), "55P03" | "40P01" | "40001")
@@ -50,6 +46,35 @@ pub(super) fn transient_reason(error: &RuntimeError) -> Option<&'static str> {
     }
 }
 
+/// MySQL server and client error numbers that describe a passing backend
+/// condition rather than a request fault. Beyond lock waits and lost
+/// connections, an InnoDB Cluster primary switch behind MySQL Router can
+/// surface: 1053 server shutdown in progress, 1290 a write reaching a member
+/// that is (now) `super_read_only`, 1927 connection killed, 2055 connection
+/// lost, 3100 a Group Replication commit hook refusal while the member has no
+/// quorum, 4031 a server-side disconnect; and 3101 a transaction rolled back
+/// at commit by group certification, which a retry can pass.
+pub(super) fn mysql_transient_reason(number: u16) -> Option<&'static str> {
+    match number {
+        1205 | 1213 | 3101 => Some("backend_lock_pressure"),
+        1053 | 1290 | 1927 | 2002 | 2003 | 2006 | 2013 | 2055 | 3100 | 4031 => Some("backend_unavailable"),
+        _ => None,
+    }
+}
+
+#[test]
+fn mysql_cluster_failover_errors_are_transient() {
+    for number in [1053, 1290, 1927, 2002, 2003, 2006, 2013, 2055, 3100, 4031] {
+        assert_eq!(mysql_transient_reason(number), Some("backend_unavailable"), "{number}");
+    }
+    for number in [1205, 1213, 3101] {
+        assert_eq!(mysql_transient_reason(number), Some("backend_lock_pressure"), "{number}");
+    }
+    for number in [1062, 1146, 1452] {
+        assert_eq!(mysql_transient_reason(number), None, "{number}");
+    }
+}
+
 fn failure(status: StatusCode, reason: &'static str, worker_id: &str, token: &str) -> RuntimeError {
     RuntimeError::Protocol {
         status,

rust/tests/execution/activity_results.rs

@@ -354,3 +354,48 @@ async fn activity_result_read_uses_canonical_retry_failure_and_cancellation_outc
     runtime.close().await;
     database.remove().await;
 }
+
+/// PHP's activity poll answer carries `server_time` (since 2.11.0), leased
+/// or empty. Published SDKs measure the lease against it and refuse a claim
+/// without it: Python SDK 2.12.2 raised `ActivityLeaseAuthorityLost`
+/// (`invalid_activity_claim`) for every activity native delivered.
+#[tokio::test]
+async fn activity_poll_answers_carry_the_server_clock() {
+    let database = TestDatabase::new().await;
+    let runtime = database.open().await.unwrap();
+    let app = router(runtime.clone());
+    let micros = |value: &Value| {
+        let text = value.as_str().unwrap_or_default();
+        chrono::DateTime::parse_from_rfc3339(text).is_ok()
+            && text.ends_with('Z')
+            && text.rsplit_once('.').is_some_and(|(_, fraction)| fraction.len() == 7)
+    };
+    let app = &app;
+    let call = |method, path: String, body| async move {
+        namespaces::scoped(app, "default", method, &path, body).await
+    };
+    let poll = || call("POST", "/api/worker/activity-tasks/poll".into(),
+        json!({"worker_id":"clock-worker","task_queue":"test","timeout_seconds":0}));
+    let registered = call("POST", "/api/worker/register".into(), json!({
+        "worker_id":"clock-worker","task_queue":"test","runtime":"python",
+        "supported_workflow_types":["echo"],"supported_activity_types":["clock-activity"]})).await;
+    assert_eq!(registered.0, StatusCode::CREATED, "{}", registered.1);
+    namespaces::start_scoped(&app, "default", "clock-run").await;
+    let workflow = call("POST", "/api/worker/workflow-tasks/poll".into(),
+        json!({"worker_id":"clock-worker","task_queue":"test","timeout_seconds":0})).await.1["task"].clone();
+    let done = call("POST", format!("/api/worker/workflow-tasks/{}/complete", workflow["task_id"].as_str().unwrap()),
+        completion(&workflow, json!([{"type":"schedule_activity","activity_type":"clock-activity",
+            "arguments":envelope(Payload::Array(vec![])),"start_to_close_timeout":120}]))).await;
+    assert_eq!(done.0, StatusCode::OK, "{}", done.1);
+    let leased = poll().await;
+    assert_eq!(leased.0, StatusCode::OK, "{}", leased.1);
+    assert_eq!(leased.1["poll_status"], "leased", "{}", leased.1);
+    assert!(micros(&leased.1["server_time"]), "{}", leased.1);
+    let expiry = chrono::DateTime::parse_from_rfc3339(leased.1["task"]["lease_expires_at"].as_str().unwrap()).unwrap();
+    let clock = chrono::DateTime::parse_from_rfc3339(leased.1["server_time"].as_str().unwrap()).unwrap();
+    assert!(expiry > clock, "{}", leased.1);
+    let empty = poll().await;
+    assert_eq!(empty.0, StatusCode::OK, "{}", empty.1);
+    assert_eq!(empty.1["poll_status"], "empty", "{}", empty.1);
+    assert!(micros(&empty.1["server_time"]), "{}", empty.1);
+}

scripts/perf/mysql-innodb-cluster/README.md

@@ -0,0 +1,42 @@
+# Native behind MySQL InnoDB Cluster and MySQL Router
+
+A disposable harness that qualifies the native server on a highly available
+MySQL topology: a three-member single-primary MySQL 8.4 InnoDB Cluster behind MySQL
+Router, with native connected to Router's classic read-write port. Results and
+the settings they support are in
+[the November 1 checkpoint](../../../docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md).
+
+Everything runs in Docker on one host: three members at 1 GiB each, Router at
+256 MiB, native at 512 MiB. `up.sh` generates a random password into
+`$DW_MRC_STATE/mysql-password`; scripts pass it through the environment.
+
+```sh
+export DW_MRC_STATE=$PWD/.mrc DW_NATIVE_BINARY=/abs/path/durable-workflow-server
+mkdir -p "$DW_MRC_STATE"   # then put durable-workflow 2.12.2 under $DW_MRC_STATE/py
+bash up.sh                                    # cluster + Router on network dw-mrc
+bash failover.sh run2 dwrun2 150 30 kill      # hard kill the primary 30 s in
+r=$DW_MRC_STATE/run2; bash dbcheck.sh dwrun2 > "$r/db.tsv"
+{ echo '#kill'; cat "$r/kill.json"; echo '#load'; cat "$r/load.jsonl"
+  echo '#probe'; cat "$r/probe.jsonl"; echo '#db'; cat "$r/db.tsv"; } | python3 verify.py
+RATE=1 bash failover.sh run4 dwrun4 90 30 pause   # hung primary
+```
+
+The load driver imports the published Python SDK, `durable-workflow` 2.12.2,
+from `$DW_MRC_STATE/py` (a `--target` install for CPython 3.12; use the
+repository's patched-pip bootstrap, `scripts/ci/bootstrap-patched-pip.sh`).
+
+`failover.sh` starts native on a fresh database through Router, probes
+`/api/ready` and `/api/health` every 0.5 s, drives Python SDK workflows (a
+1 s timer then two activities), and fails the current primary. `kill` is a
+crash, `stop` a clean shutdown, `pause` a hung host. Restart or unpause the
+member afterwards; it rejoins on its own. `verify.py` reports outcomes,
+readiness through the failover, and database integrity: one completion per
+run, exact history counts, no duplicate history rows, no leftover leases.
+
+Members get fixed addresses and static host entries, so a dead member's name
+still resolves, as it does on real hosts. Docker's embedded DNS drops a killed
+container's name, and the members then lost quorum entirely (see the
+checkpoint).
+
+Tear down: `docker rm -f -v dw-mrc-native dw-mrc-router dw-mrc-db1 dw-mrc-db2
+dw-mrc-db3 && docker network rm dw-mrc`.

scripts/perf/mysql-innodb-cluster/dbcheck.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Database integrity after a failover run. Usage: dbcheck.sh <database> > rows.tsv
+set -uo pipefail
+db=$1
+[[ "$db" =~ ^[A-Za-z0-9_]+$ ]] || { echo "database must be a plain name" >&2; exit 2; }
+M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7
+MYSQL_PWD=$(cat "${DW_MRC_STATE:?set DW_MRC_STATE}/mysql-password")
+export MYSQL_PWD
+timeout 60 docker run --rm --network dw-mrc --memory 128m -e MYSQL_PWD "$M" mysql -h dw-mrc-router -P 6446 -uroot -N "$db" -e "
+SELECT 'runs by status', GROUP_CONCAT(CONCAT(status,'=',n)) FROM (SELECT status, COUNT(*) n FROM workflow_runs GROUP BY status) x;
+SELECT 'workflow_tasks open', COUNT(*) FROM workflow_tasks WHERE status IN ('ready','leased');
+SELECT 'activity_attempts running', COUNT(*) FROM activity_attempts WHERE status='running';
+SELECT 'runs with !=1 WorkflowCompleted (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='WorkflowCompleted')<>1;
+SELECT 'runs with !=2 ActivityCompleted (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='ActivityCompleted')<>2;
+SELECT 'runs with !=1 TimerFired (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='TimerFired')<>1;
+SELECT 'duplicate (run,sequence) history rows', COUNT(*) FROM (SELECT workflow_run_id, sequence FROM workflow_history_events GROUP BY 1,2 HAVING COUNT(*)>1) d;
+SELECT 'history events by type', GROUP_CONCAT(CONCAT(event_type,'=',n) ORDER BY event_type) FROM (SELECT event_type, COUNT(*) n FROM workflow_history_events GROUP BY 1) y;
+" 2>/dev/null | sed 's/\t/: /'