From 19c459bae5a76bd199a84d954d8a7ca48b04465e Mon Sep 17 00:00:00 2001
From: Richard McDaniel <1130888+rmcdaniel@users.noreply.github.com>
Date: Sun, 1 Nov 2026 21:23:44 +0000
Subject: [PATCH 1/3] Rust #325: qualify native on MySQL InnoDB Cluster behind
 Router

Native ran on a three-member single-primary MySQL 8.4 InnoDB Cluster
through Router's read-write port: schema creation, the timer and activity
lifecycle, and primary crash, clean shutdown and hang. History stayed
exact with no leftover leases. Native answered 503 while no primary was
reachable and recovered without a restart, 1 to 5 s after Router switched.

Two defects found:
- The activity poll answer lacked PHP's top-level server_time (published
  since 2.11.0). Python SDK 2.12.2 measures a claimed lease against it and
  refused every activity native delivered, on any database.
- MySQL errors a primary switch can raise (1053, 1290, 1927, 2055, 3100,
  4031; 3101 as lock pressure) are now classified transient.

The harness is in scripts/perf/mysql-innodb-cluster; results and operator
settings are in the November 1 checkpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../2026-11-01-mysql-innodb-cluster-router.md | 115 ++++++++++++++++++
 docs/rust-server-port.md                      |  12 ++
 rust/src/runtime/store.rs                     |  21 +++-
 rust/src/runtime/worker_incarnations.rs       |  35 +++++-
 rust/tests/execution/activity_results.rs      |  45 +++++++
 scripts/perf/mysql-innodb-cluster/README.md   |  38 ++++++
 scripts/perf/mysql-innodb-cluster/dbcheck.sh  |  14 +++
 scripts/perf/mysql-innodb-cluster/failover.sh |  54 ++++++++
 scripts/perf/mysql-innodb-cluster/load.py     |  82 +++++++++++++
 scripts/perf/mysql-innodb-cluster/up.sh       |  37 ++++++
 scripts/perf/mysql-innodb-cluster/verify.py   |  35 ++++++
 11 files changed, 479 insertions(+), 9 deletions(-)
 create mode 100644 docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md
 create mode 100644 scripts/perf/mysql-innodb-cluster/README.md
 create mode 100755 scripts/perf/mysql-innodb-cluster/dbcheck.sh
 create mode 100755 scripts/perf/mysql-innodb-cluster/failover.sh
 create mode 100755 scripts/perf/mysql-innodb-cluster/load.py
 create mode 100755 scripts/perf/mysql-innodb-cluster/up.sh
 create mode 100755 scripts/perf/mysql-innodb-cluster/verify.py

diff --git a/docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md b/docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md
new file mode 100644
index 000000000..b3b768457
--- /dev/null
+++ b/docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md
@@ -0,0 +1,115 @@
+# Native on MySQL InnoDB Cluster behind MySQL Router (November 1)
+
+A highly available native deployment on MySQL runs native 3.x on a MySQL 8
+InnoDB Cluster behind MySQL Router. This checkpoint qualifies that topology
+on one disposable host with the
+[harness](../../scripts/perf/mysql-innodb-cluster/README.md). It is evidence
+for the topology and its failure handling, not a capacity measurement.
+
+## Conditions
+
+- MySQL 8.4.13 (`mirror.gcr.io/library/mysql@sha256:a0225d1e…`), three
+  members, single primary, default Group Replication settings (MySQL
+  communication stack, `group_replication_member_expel_timeout` 5 s), 1 GiB
+  and a 128 MiB buffer pool each.
+- MySQL Router 8.4.10
+  (`container-registry.oracle.com/mysql/community-router@sha256:d704471c…`),
+  bootstrapped defaults; native uses the classic read-write port 6446.
+- Native from this branch (main at `bbed2bec` plus the two fixes below), with
+  `DW_RUST_MAINTENANCE_LOOP=1`, 512 MiB.
+- Load: published Python SDK 2.12.2, one worker (eight workflow and eight
+  activity slots), workflows of a 1 s timer then two activities.
+
+## Results
+
+| Run | Failure | Workflows completed | Starts refused (503) | Readiness non-200 | Integrity |
+| --- | --- | --- | --- | --- | --- |
+| Baseline | none | 60/60, p50 5.1 s | 0 | none | exact |
+| 2 | primary crash (`docker kill`) | 106 of 107 started; the 107th made ready again at its activity lease expiry | 43 | +4.4 s to +40.7 s | exact |
+| 3 | primary clean shutdown (`docker stop`) | 150/150 | 0 | one sample, ready again by +3.8 s | exact |
+| 4 | primary hung (`docker pause`) | 74/74 | 16 | +0.5 s to +32.7 s | exact |
+
+"Exact" means: every completed run has one `WorkflowCompleted`, one
+`TimerFired` and two `ActivityCompleted` events, no `(run, sequence)` history
+row is duplicated, and no workflow task or activity attempt is left leased.
+
+- **Schema creation through Router** works: native created and verified its
+  schema on a fresh database through port 6446.
+- **Crash recovery time** (run 2, measured from the kill): Group Replication
+  elected the new primary at +23.4 s, Router routed writes to it at +36 s,
+  native's health answered 200 at +37.0 s and readiness at +40.7 s. Native
+  adds 1 to 5 s after Router switches; the rest is election and Router's
+  metadata refresh, which operators can tune.
+- **Honest answers during the window**: with no primary, every request,
+  including starts, answers `503 storage_unavailable`; readiness answers 503
+  and health answers 503 or times out at the probe. Nothing answers 200 while
+  writes cannot commit. Native reconnects without a restart; the scheduler
+  loop logs `runtime_scheduler_storage_unavailable` once and resumes.
+- **In-flight work**: an activity claimed 0.4 s before the crash could not be
+  completed by its worker during the window. At its lease expiry native
+  expired the attempt, failed the activity under its retry policy and made a
+  workflow task ready, so the run resumes; nothing stays leased.
+- **Client-visible errors**: the Python SDK's `start_workflow` and
+  `result()` do not retry 503, so a caller sees `storage_unavailable` for
+  calls made during the window (19 `result()` calls in run 2, whose runs did
+  complete server-side). Callers should retry 503 with backoff.
+- **Throughput is the driver's limit, not the failover's**: at 2 starts per
+  second the single SDK worker saturates (a no-failover control had a 73 s
+  median); run 4 used 1 per second.
+
+A first crash run used Docker's embedded DNS, which drops a killed
+container's name. The surviving members, on the MySQL communication stack,
+then reported each other `UNREACHABLE` intermittently and lost quorum; no
+primary was elected and the old one could not rejoin. With fixed addresses
+and static host entries, as real hosts have, the same crash failed over
+cleanly. Cluster members need names that keep resolving while a member is
+down.
+
+## Defects found and fixed
+
+- **Activity poll without `server_time`** (any topology). Published PHP has
+  answered `POST /api/worker/activity-tasks/poll` with a top-level
+  `server_time` since 2.11.0; native did not. Python SDK 2.12.2 measures a
+  claimed activity's lease against it and refused every activity native
+  delivered (`ActivityLeaseAuthorityLost`, `invalid_activity_claim`), so no
+  activity could run under that SDK. Native now adds `server_time` to every
+  activity poll answer, leased or empty
+  (`activity_poll_answers_carry_the_server_clock`).
+- **Failover error classes**. MySQL errors a primary switch can surface are
+  now classified transient: `backend_unavailable` for 1053, 1290, 1927, 2055,
+  3100 and 4031, `backend_lock_pressure` for 3101
+  (`mysql_cluster_failover_errors_are_transient`). Every database error
+  already answered 503; the class drives the bounded diagnostic and the
+  retryable worker deregistration receipt.
+
+## Settings for operators
+
+- **Connection**: `DB_CONNECTION=mysql`, `DB_HOST=<router>`, `DB_PORT=6446`
+  (Router's classic read-write port, never the read-only 6447 or the
+  read/write-split 6450), `DB_DATABASE`, `DB_USERNAME`, `DB_PASSWORD`, plus
+  `DB_SSLMODE`/`DB_SSLROOTCERT` for TLS to Router.
+- **Pool** (fixed in native): four connections per process, 5 s acquire
+  timeout, connections tested before reuse, `lock_wait_timeout` and
+  `innodb_lock_wait_timeout` of 5 s per session. Size Router's
+  `max_connections` for four per native process plus headroom.
+- **Probes**: readiness gates traffic. While no primary is reachable, health
+  and readiness can take up to about 5 s to answer 503 (the pool's acquire
+  timeout), so give probes a timeout of at least 6 s. Do not restart native on
+  a database outage: health is 503 for the whole failover window and a
+  restart does not shorten it. Expect 35 to 45 s of 503 after a primary crash
+  with default Group Replication and Router settings, and a few seconds after
+  a clean shutdown.
+- **Router**: keep `connection_sharing` off (the default); native sets
+  session variables on connect. Router's metadata `ttl` and Group Replication
+  notifications, and Group Replication's `member_expel_timeout`, set the
+  failover window.
+- **Named locks**: native takes MySQL `GET_LOCK` only while bootstrapping or
+  upgrading its schema and in upgrade takeover, never at runtime. Named locks
+  are local to the member, so a failover during a schema upgrade releases the
+  lock; run upgrades with one native process.
+
+## Not covered
+
+Multi-primary mode, Router read/write split, Router on a separate host or
+several Routers, long-running load, cross-host network partitions, TLS to
+Router, and the Rust and PHP SDKs.
diff --git a/docs/rust-server-port.md b/docs/rust-server-port.md
index 85cd9aefd..d64b1a804 100644
--- a/docs/rust-server-port.md
+++ b/docs/rust-server-port.md
@@ -272,6 +272,18 @@ These are confirmed differences from the published PHP reference and open
 contract decisions found while qualifying native slices. Each one needs a
 contract fixture before its correction.
 
+- **MySQL InnoDB Cluster behind MySQL Router** (November 1, qualified on one
+  host; [checkpoint](rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md)).
+  Native runs on a single-primary InnoDB Cluster through Router's
+  read-write port. Schema creation, the workflow, timer and
+  activity lifecycle, and primary crash, clean shutdown and hang all pass
+  with exact history and no leftover leases; native answers 503 while no
+  primary is reachable and recovers without a restart, 1 to 5 s after Router
+  switches. Two defects found and fixed: the activity poll answer lacked
+  PHP's `server_time` (since 2.11.0), so Python SDK 2.12.2 refused every
+  activity native delivered on any database; and MySQL failover errors
+  (1053, 1290, 1927, 2055, 3100, 3101, 4031) are now classified transient.
+  Multi-primary, read/write split and several Routers are not covered.
 - **History export waits and timeline** (November 1, closed,
   [checkpoint](rust-server-port-checkpoints/2026-11-01-history-export-waits.md)).
   Native's export now serves the run's stored waits with each kind's
diff --git a/rust/src/runtime/store.rs b/rust/src/runtime/store.rs
index f58159032..e4ce6d83e 100644
--- a/rust/src/runtime/store.rs
+++ b/rust/src/runtime/store.rs
@@ -266,6 +266,16 @@ fn after(seconds: i64) -> DateTime<Utc> {
     now() + chrono::Duration::seconds(seconds)
 }
 
+/// As PHP's `ActivityTaskController::poll` (since 2.11.0): an activity poll
+/// answer carries `server_time`, the clock its lease expiry is measured
+/// against. Published SDKs refuse a claimed activity without it.
+fn activity_server_time(kind: &str, mut answer: Value) -> Value {
+    if kind == "activity" {
+        answer["server_time"] = json!(super::activity_lease_renewal::micros(now()));
+    }
+    answer
+}
+
 impl<DB: Backend> Store<DB>
 where
     for<'c> &'c mut DB::Connection: Executor<'c, Database = DB>,
@@ -1974,19 +1984,22 @@ where
                 // Compressed after the claim commits; a replayed receipt keeps
                 // the plain events and takes this request's encoding.
                 super::history_pages::compress(&mut task, encoding)?;
-                return Ok(
+                return Ok(activity_server_time(
+                    kind,
                     json!({"task": task, "poll_status": "leased", "server_capabilities": capabilities()}),
-                );
+                ));
             }
             if tokio::time::Instant::now() >= deadline {
-                return self.no_task_answer(&body, poll_request.as_ref(), "empty", encoding).await;
+                return self.no_task_answer(&body, poll_request.as_ref(), "empty", encoding).await
+                    .map(|answer| activity_server_time(kind, answer));
             }
             if self
                 .poll_queue_changed(&body, kind, &mut cross_kind)
                 .await?
                 && tokio::time::Instant::now() < deadline
             {
-                return self.no_task_answer(&body, poll_request.as_ref(), "task_queue_changed", encoding).await;
+                return self.no_task_answer(&body, poll_request.as_ref(), "task_queue_changed", encoding).await
+                    .map(|answer| activity_server_time(kind, answer));
             }
             #[cfg(debug_assertions)]
             if let Some(request) = &poll_request {
diff --git a/rust/src/runtime/worker_incarnations.rs b/rust/src/runtime/worker_incarnations.rs
index 6335e3737..2100717f3 100644
--- a/rust/src/runtime/worker_incarnations.rs
+++ b/rust/src/runtime/worker_incarnations.rs
@@ -23,11 +23,7 @@ pub(super) fn transient_reason(error: &RuntimeError) -> Option<&'static str> {
         ) => Some("backend_unavailable"),
         RuntimeError::Database(sqlx::Error::Database(error)) => {
             if let Some(mysql) = error.try_downcast_ref::<sqlx::mysql::MySqlDatabaseError>() {
-                return match mysql.number() {
-                    1205 | 1213 => Some("backend_lock_pressure"),
-                    2002 | 2003 | 2006 | 2013 => Some("backend_unavailable"),
-                    _ => None,
-                };
+                return mysql_transient_reason(mysql.number());
             }
             let code = error.code()?;
             if matches!(code.as_ref(), "55P03" | "40P01" | "40001")
@@ -50,6 +46,35 @@ pub(super) fn transient_reason(error: &RuntimeError) -> Option<&'static str> {
     }
 }
 
+/// MySQL server and client error numbers that describe a passing backend
+/// condition rather than a request fault. Beyond lock waits and lost
+/// connections, an InnoDB Cluster primary switch behind MySQL Router can
+/// surface: 1053 server shutdown in progress, 1290 a write reaching a member
+/// that is (now) `super_read_only`, 1927 connection killed, 2055 connection
+/// lost, 3100 a Group Replication commit hook refusal while the member has no
+/// quorum, 4031 a server-side disconnect; and 3101 a transaction rolled back
+/// at commit by group certification, which a retry can pass.
+pub(super) fn mysql_transient_reason(number: u16) -> Option<&'static str> {
+    match number {
+        1205 | 1213 | 3101 => Some("backend_lock_pressure"),
+        1053 | 1290 | 1927 | 2002 | 2003 | 2006 | 2013 | 2055 | 3100 | 4031 => Some("backend_unavailable"),
+        _ => None,
+    }
+}
+
+#[test]
+fn mysql_cluster_failover_errors_are_transient() {
+    for number in [1053, 1290, 1927, 2002, 2003, 2006, 2013, 2055, 3100, 4031] {
+        assert_eq!(mysql_transient_reason(number), Some("backend_unavailable"), "{number}");
+    }
+    for number in [1205, 1213, 3101] {
+        assert_eq!(mysql_transient_reason(number), Some("backend_lock_pressure"), "{number}");
+    }
+    for number in [1062, 1146, 1452] {
+        assert_eq!(mysql_transient_reason(number), None, "{number}");
+    }
+}
+
 fn failure(status: StatusCode, reason: &'static str, worker_id: &str, token: &str) -> RuntimeError {
     RuntimeError::Protocol {
         status,
diff --git a/rust/tests/execution/activity_results.rs b/rust/tests/execution/activity_results.rs
index d8b2657ba..2c9270943 100644
--- a/rust/tests/execution/activity_results.rs
+++ b/rust/tests/execution/activity_results.rs
@@ -354,3 +354,48 @@ async fn activity_result_read_uses_canonical_retry_failure_and_cancellation_outc
     runtime.close().await;
     database.remove().await;
 }
+
+/// PHP's activity poll answer carries `server_time` (since 2.11.0), leased
+/// or empty. Published SDKs measure the lease against it and refuse a claim
+/// without it: Python SDK 2.12.2 raised `ActivityLeaseAuthorityLost`
+/// (`invalid_activity_claim`) for every activity native delivered.
+#[tokio::test]
+async fn activity_poll_answers_carry_the_server_clock() {
+    let database = TestDatabase::new().await;
+    let runtime = database.open().await.unwrap();
+    let app = router(runtime.clone());
+    let micros = |value: &Value| {
+        let text = value.as_str().unwrap_or_default();
+        chrono::DateTime::parse_from_rfc3339(text).is_ok()
+            && text.ends_with('Z')
+            && text.rsplit_once('.').is_some_and(|(_, fraction)| fraction.len() == 7)
+    };
+    let app = &app;
+    let call = |method, path: String, body| async move {
+        namespaces::scoped(app, "default", method, &path, body).await
+    };
+    let poll = || call("POST", "/api/worker/activity-tasks/poll".into(),
+        json!({"worker_id":"clock-worker","task_queue":"test","timeout_seconds":0}));
+    let registered = call("POST", "/api/worker/register".into(), json!({
+        "worker_id":"clock-worker","task_queue":"test","runtime":"python",
+        "supported_workflow_types":["echo"],"supported_activity_types":["clock-activity"]})).await;
+    assert_eq!(registered.0, StatusCode::CREATED, "{}", registered.1);
+    namespaces::start_scoped(&app, "default", "clock-run").await;
+    let workflow = call("POST", "/api/worker/workflow-tasks/poll".into(),
+        json!({"worker_id":"clock-worker","task_queue":"test","timeout_seconds":0})).await.1["task"].clone();
+    let done = call("POST", format!("/api/worker/workflow-tasks/{}/complete", workflow["task_id"].as_str().unwrap()),
+        completion(&workflow, json!([{"type":"schedule_activity","activity_type":"clock-activity",
+            "arguments":envelope(Payload::Array(vec![])),"start_to_close_timeout":120}]))).await;
+    assert_eq!(done.0, StatusCode::OK, "{}", done.1);
+    let leased = poll().await;
+    assert_eq!(leased.0, StatusCode::OK, "{}", leased.1);
+    assert_eq!(leased.1["poll_status"], "leased", "{}", leased.1);
+    assert!(micros(&leased.1["server_time"]), "{}", leased.1);
+    let expiry = chrono::DateTime::parse_from_rfc3339(leased.1["task"]["lease_expires_at"].as_str().unwrap()).unwrap();
+    let clock = chrono::DateTime::parse_from_rfc3339(leased.1["server_time"].as_str().unwrap()).unwrap();
+    assert!(expiry > clock, "{}", leased.1);
+    let empty = poll().await;
+    assert_eq!(empty.0, StatusCode::OK, "{}", empty.1);
+    assert_eq!(empty.1["poll_status"], "empty", "{}", empty.1);
+    assert!(micros(&empty.1["server_time"]), "{}", empty.1);
+}
diff --git a/scripts/perf/mysql-innodb-cluster/README.md b/scripts/perf/mysql-innodb-cluster/README.md
new file mode 100644
index 000000000..efdc572e2
--- /dev/null
+++ b/scripts/perf/mysql-innodb-cluster/README.md
@@ -0,0 +1,38 @@
+# Native behind MySQL InnoDB Cluster and MySQL Router
+
+A disposable harness that qualifies the native server on a highly available
+MySQL topology: a three-member single-primary MySQL 8.4 InnoDB Cluster behind MySQL
+Router, with native connected to Router's classic read-write port. Results and
+the settings they support are in
+[the November 1 checkpoint](../../../docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md).
+
+Everything runs in Docker on one host: three members at 1 GiB each, Router at
+256 MiB, native at 512 MiB. Passwords are disposable fixture values.
+
+```sh
+export DW_MRC_STATE=$PWD/.mrc DW_NATIVE_BINARY=/abs/path/durable-workflow-server
+mkdir -p "$DW_MRC_STATE"
+docker run --rm --user 1000:1000 -e HOME=/tmp -v "$DW_MRC_STATE:/mrc" python:3.12-slim \
+  pip install -q --target /mrc/py durable-workflow==2.12.2
+bash up.sh                                    # cluster + Router on network dw-mrc
+bash failover.sh run2 dwrun2 150 30 kill      # hard kill the primary 30 s in
+bash dbcheck.sh dwrun2 > "$DW_MRC_STATE/run2/db.tsv"
+python3 verify.py "$DW_MRC_STATE/run2" "$DW_MRC_STATE/run2/db.tsv"
+RATE=1 bash failover.sh run4 dwrun4 90 30 pause   # hung primary
+```
+
+`failover.sh` starts native on a fresh database through Router, probes
+`/api/ready` and `/api/health` every 0.5 s, drives Python SDK workflows (a
+1 s timer then two activities), and fails the current primary. `kill` is a
+crash, `stop` a clean shutdown, `pause` a hung host. Restart or unpause the
+member afterwards; it rejoins on its own. `verify.py` reports outcomes,
+readiness through the failover, and database integrity: one completion per
+run, exact history counts, no duplicate history rows, no leftover leases.
+
+Members get fixed addresses and static host entries, so a dead member's name
+still resolves, as it does on real hosts. Docker's embedded DNS drops a killed
+container's name, and the members then lost quorum entirely (see the
+checkpoint).
+
+Tear down: `docker rm -f -v dw-mrc-native dw-mrc-router dw-mrc-db1 dw-mrc-db2
+dw-mrc-db3 && docker network rm dw-mrc`.
diff --git a/scripts/perf/mysql-innodb-cluster/dbcheck.sh b/scripts/perf/mysql-innodb-cluster/dbcheck.sh
new file mode 100755
index 000000000..1323b4927
--- /dev/null
+++ b/scripts/perf/mysql-innodb-cluster/dbcheck.sh
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Database integrity after a failover run. Usage: dbcheck.sh <database> > rows.tsv
+db=$1
+M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7
+timeout 60 docker run --rm --network dw-mrc --memory 128m $M mysql -h dw-mrc-router -P 6446 -uroot -pdwfixture -N $db -e "
+SELECT 'runs by status', GROUP_CONCAT(CONCAT(status,'=',n)) FROM (SELECT status, COUNT(*) n FROM workflow_runs GROUP BY status) x;
+SELECT 'workflow_tasks open', COUNT(*) FROM workflow_tasks WHERE status IN ('ready','leased');
+SELECT 'activity_attempts running', COUNT(*) FROM activity_attempts WHERE status='running';
+SELECT 'runs with !=1 WorkflowCompleted (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='WorkflowCompleted')<>1;
+SELECT 'runs with !=2 ActivityCompleted (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='ActivityCompleted')<>2;
+SELECT 'runs with !=1 TimerFired (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='TimerFired')<>1;
+SELECT 'duplicate (run,sequence) history rows', COUNT(*) FROM (SELECT workflow_run_id, sequence FROM workflow_history_events GROUP BY 1,2 HAVING COUNT(*)>1) d;
+SELECT 'history events by type', GROUP_CONCAT(CONCAT(event_type,'=',n) ORDER BY event_type) FROM (SELECT event_type, COUNT(*) n FROM workflow_history_events GROUP BY 1) y;
+" 2>/dev/null | sed 's/\t/: /'
diff --git a/scripts/perf/mysql-innodb-cluster/failover.sh b/scripts/perf/mysql-innodb-cluster/failover.sh
new file mode 100755
index 000000000..9b3b76953
--- /dev/null
+++ b/scripts/perf/mysql-innodb-cluster/failover.sh
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+# One primary-failover experiment against native behind MySQL Router.
+# Usage: failover.sh <run-name> <database> <count> <fail-after-seconds> [kill|stop|pause]
+# Env: DW_MRC_STATE (scratch dir; Python SDK installed under $DW_MRC_STATE/py),
+# DW_NATIVE_BINARY, RATE (workflow starts per second, default 2).
+set -uo pipefail
+run=$1 db=$2 count=$3 kill_after=$4 mode=${5:-kill}
+here=${DW_MRC_STATE:?set DW_MRC_STATE to a scratch directory}
+scripts=$(cd "$(dirname "$0")" && pwd)
+bin=${DW_NATIVE_BINARY:?set DW_NATIVE_BINARY to a native server binary built for glibc 2.39}
+M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7
+out=$here/$run; mkdir -p "$out"
+sql() { timeout 30 docker run --rm --network dw-mrc --memory 128m $M mysql -h dw-mrc-router -P 6446 -uroot -pdwfixture -N -e "$1" 2>/dev/null; }
+
+sql "CREATE DATABASE IF NOT EXISTS $db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT ALL ON $db.* TO 'dw'@'%';"
+timeout 30 docker rm -f dw-mrc-native >/dev/null 2>&1
+timeout 60 docker run -d --name dw-mrc-native --hostname dw-mrc-native --network dw-mrc --memory 512m --user 1000:1000 \
+  -v $bin:/usr/local/bin/dw-native:ro -e DW_RUST_EXPERIMENTAL=1 -e DW_RUST_MAINTENANCE_LOOP=1 -e DW_BIND_ADDRESS=0.0.0.0:8080 \
+  -e DB_CONNECTION=mysql -e DB_HOST=dw-mrc-router -e DB_PORT=6446 -e DB_DATABASE=$db -e DB_USERNAME=dw -e DB_PASSWORD=dwfixture \
+  -e DW_AUTH_TOKEN=dwtoken mirror.gcr.io/library/ubuntu@sha256:fe81560990fae86dc4743202b2cb4d3df98d0bcf949173cecef15ce1127ef852 /usr/local/bin/dw-native >/dev/null
+timeout 60 bash -c 'until docker run --rm --network dw-mrc --memory 64m mirror.gcr.io/library/python@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 python -c "import urllib.request;urllib.request.urlopen(\"http://dw-mrc-native:8080/api/ready\",timeout=2)" >/dev/null 2>&1; do sleep 1; done'
+
+# Readiness/health probe every 0.5 s.
+timeout 30 docker rm -f dw-mrc-probe >/dev/null 2>&1
+timeout 60 docker run -d --name dw-mrc-probe --network dw-mrc --memory 64m mirror.gcr.io/library/python@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 python -u -c '
+import urllib.request, json, time
+while True:
+    row = {"t": round(time.time(), 2)}
+    for p in ("ready", "health"):
+        try:
+            r = urllib.request.urlopen("http://dw-mrc-native:8080/api/" + p, timeout=2); row[p] = r.status
+        except Exception as e:
+            row[p] = getattr(e, "code", type(e).__name__)
+    print(json.dumps(row), flush=True); time.sleep(0.5)' >/dev/null
+
+# Load.
+timeout 30 docker rm -f dw-mrc-load >/dev/null 2>&1
+timeout 60 docker run -d --name dw-mrc-load --network dw-mrc --memory 512m --user 1000:1000 -e HOME=/tmp -e PYTHONPATH=/mrc/py \
+  -v $here:/mrc -v $scripts:/scripts:ro mirror.gcr.io/library/python@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 python /scripts/load.py --base-url http://dw-mrc-native:8080 --token dwtoken \
+  --count $count --rate ${RATE:-2} >/dev/null
+sleep "$kill_after"
+primary=$(sql "SELECT MEMBER_HOST FROM performance_schema.replication_group_members WHERE MEMBER_ROLE='PRIMARY'")
+echo "{\"event\":\"kill\",\"mode\":\"$mode\",\"primary\":\"$primary\",\"t\":$(date +%s.%N | cut -c1-14)}" > "$out/kill.json"
+case "$mode" in
+  kill) timeout 30 docker kill "$primary" >/dev/null ;;
+  stop) timeout 60 docker stop -t 20 "$primary" >/dev/null ;;
+  pause) timeout 30 docker pause "$primary" >/dev/null; echo "$primary" > "$out/paused" ;;
+esac
+timeout 900 bash -c 'until [ "$(docker inspect -f {{.State.Status}} dw-mrc-load 2>/dev/null)" = exited ]; do sleep 5; done'
+timeout 30 docker logs dw-mrc-load > "$out/load.jsonl" 2> "$out/load.err"
+timeout 30 docker logs dw-mrc-probe > "$out/probe.jsonl" 2>/dev/null
+timeout 30 docker logs dw-mrc-native > "$out/native.log" 2>&1
+timeout 30 docker rm -f dw-mrc-probe dw-mrc-load >/dev/null
+echo "$primary" > "$out/killed-primary"
diff --git a/scripts/perf/mysql-innodb-cluster/load.py b/scripts/perf/mysql-innodb-cluster/load.py
new file mode 100755
index 000000000..4e9eb3728
--- /dev/null
+++ b/scripts/perf/mysql-innodb-cluster/load.py
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+"""HA load for native behind MySQL Router: start workflows continuously,
+wait for each result, and record outcomes and timing as JSON lines."""
+
+import argparse
+import asyncio
+import json
+import logging
+import sys
+import time
+import uuid
+
+from durable_workflow import Client, Worker, activity, workflow
+
+import os
+logging.basicConfig(level=os.environ.get("LOGLEVEL", "ERROR"))
+logging.getLogger("durable_workflow.worker").setLevel(os.environ.get("LOGLEVEL", "ERROR"))
+WF = "ha.timer_activity"
+ACT = "ha.echo"
+
+
+@activity.defn(name=ACT)
+async def echo(value):
+    await asyncio.sleep(0.05)
+    return value
+
+
+@workflow.defn(name=WF)
+class TimerActivity:
+    def run(self, context, request):
+        yield context.sleep(1)
+        first = yield context.schedule_activity(ACT, [{"step": 1, "n": request["n"]}])
+        second = yield context.schedule_activity(ACT, [{"step": 2, "n": request["n"]}])
+        return {"n": request["n"], "first": first, "second": second}
+
+
+def emit(record):
+    record["t"] = round(time.time(), 3)
+    sys.stdout.write(json.dumps(record) + "\n")
+    sys.stdout.flush()
+
+
+async def one(client, n, queue):
+    started = time.monotonic()
+    wid = f"ha-{n}-{uuid.uuid4().hex[:8]}"
+    try:
+        handle = await client.start_workflow(workflow_type=WF, task_queue=queue, workflow_id=wid, input=[{"n": n}])
+    except Exception as error:  # start refused or connection lost
+        emit({"event": "start_error", "n": n, "wid": wid, "error": f"{type(error).__name__}: {str(error)[:200]}"})
+        return
+    emit({"event": "started", "n": n, "wid": wid})
+    try:
+        result = await handle.result(timeout=300, poll_interval=0.5)
+        ok = result == {"n": n, "first": {"step": 1, "n": n}, "second": {"step": 2, "n": n}}
+        emit({"event": "completed", "n": n, "wid": wid, "ok": ok, "seconds": round(time.monotonic() - started, 3)})
+    except Exception as error:
+        emit({"event": "result_error", "n": n, "wid": wid, "error": f"{type(error).__name__}: {str(error)[:200]}"})
+
+
+async def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base-url", required=True)
+    parser.add_argument("--token", required=True)
+    parser.add_argument("--count", type=int, default=120)
+    parser.add_argument("--rate", type=float, default=2.0)
+    args = parser.parse_args()
+    queue = "ha-queue"
+    async with Client(args.base_url, token=args.token, namespace="default", timeout=10) as client:
+        worker = Worker(client, task_queue=queue, workflows=[TimerActivity], activities=[echo],
+                        worker_id=f"ha-worker-{uuid.uuid4().hex[:6]}", poll_timeout=2,
+                        max_concurrent_workflow_tasks=8, max_concurrent_activity_tasks=8, heartbeat_interval=5)
+        worker_task = asyncio.create_task(worker.run())
+        tasks = []
+        for n in range(args.count):
+            tasks.append(asyncio.create_task(one(client, n, queue)))
+            await asyncio.sleep(1.0 / args.rate)
+        await asyncio.gather(*tasks)
+        emit({"event": "done"})
+        worker_task.cancel()
+
+
+asyncio.run(main())
diff --git a/scripts/perf/mysql-innodb-cluster/up.sh b/scripts/perf/mysql-innodb-cluster/up.sh
new file mode 100755
index 000000000..2710b279e
--- /dev/null
+++ b/scripts/perf/mysql-innodb-cluster/up.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+# Disposable 3-member MySQL 8.4 InnoDB Cluster + Router with fixed IPs and
+# static host entries (names keep resolving after a member dies, as on hosts).
+set -euo pipefail
+here=${DW_MRC_STATE:?set DW_MRC_STATE to a scratch directory}
+M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7
+R=container-registry.oracle.com/mysql/community-router@sha256:d704471c2bb78fa833790dc197959c004d09af5aa18f5f153ff40e7268f54043
+hosts=(--add-host dw-mrc-db1:172.30.77.11 --add-host dw-mrc-db2:172.30.77.12 --add-host dw-mrc-db3:172.30.77.13 --add-host dw-mrc-router:172.30.77.20)
+timeout 30 docker network create --subnet 172.30.77.0/24 dw-mrc >/dev/null 2>&1 || true
+for i in 1 2 3; do
+  timeout 60 docker run -d --name dw-mrc-db$i --hostname dw-mrc-db$i --network dw-mrc --ip 172.30.77.1$i "${hosts[@]}" --memory 1g \
+    -e MYSQL_ROOT_PASSWORD=dwfixture -e MYSQL_ROOT_HOST=% $M --server-id=$i --gtid-mode=ON --enforce-gtid-consistency=ON \
+    --report-host=dw-mrc-db$i --innodb-buffer-pool-size=128M --max-connections=200 >/dev/null
+done
+for i in 1 2 3; do timeout 180 bash -c "until docker exec dw-mrc-db$i mysqladmin -uroot -pdwfixture ping --silent 2>/dev/null; do sleep 3; done"; done
+sleep 25
+cat > $here/build.js <<'EOF'
+['dw-mrc-db1','dw-mrc-db2','dw-mrc-db3'].forEach(function (h) { dba.configureInstance('root:dwfixture@' + h + ':3306', {restart: false}); });
+var c = dba.createCluster('dw');
+shell.options['dba.restartWaitTimeout'] = 300;
+c.addInstance('root:dwfixture@dw-mrc-db2:3306', {recoveryMethod: 'clone'});
+c.addInstance('root:dwfixture@dw-mrc-db3:3306', {recoveryMethod: 'clone'});
+c.rescan({addUnmanaged: true, removeObsolete: true});
+var t = c.status().defaultReplicaSet.topology; for (var k in t) print(k + ' ' + t[k].memberRole + ' ' + t[k].status + '\n');
+EOF
+timeout 30 docker cp $here/build.js dw-mrc-db1:/tmp/build.js
+# Clone restarts the joiner's mysqld; the container exits, so start it again.
+( for _ in $(seq 1 200); do for i in 2 3; do [ "$(docker inspect -f '{{.State.Status}}' dw-mrc-db$i 2>/dev/null)" = exited ] && docker start dw-mrc-db$i >/dev/null; done; sleep 3; done ) &
+watcher=$!
+timeout 900 docker exec -e LC_ALL=C dw-mrc-db1 mysqlsh --uri 'root:dwfixture@dw-mrc-db1:3306' --js --file /tmp/build.js 2>&1 | grep -E 'PRIMARY|SECONDARY|ERROR' || true
+kill $watcher 2>/dev/null || true
+timeout 60 docker exec dw-mrc-db1 mysql -uroot -pdwfixture -e "CREATE USER IF NOT EXISTS 'dw'@'%' IDENTIFIED BY 'dwfixture';" 2>/dev/null
+timeout 60 docker run -d --name dw-mrc-router --hostname dw-mrc-router --network dw-mrc --ip 172.30.77.20 "${hosts[@]}" --memory 256m \
+  -e MYSQL_HOST=dw-mrc-db1 -e MYSQL_PORT=3306 -e MYSQL_USER=root -e MYSQL_PASSWORD=dwfixture -e MYSQL_INNODB_CLUSTER_MEMBERS=3 \
+  -e MYSQL_CREATE_ROUTER_USER=0 $R mysqlrouter >/dev/null
+timeout 120 bash -c 'until docker logs dw-mrc-router 2>&1 | grep -q "listening on .0.0.0.0:6446"; do sleep 3; done'
+echo cluster-and-router-up
diff --git a/scripts/perf/mysql-innodb-cluster/verify.py b/scripts/perf/mysql-innodb-cluster/verify.py
new file mode 100755
index 000000000..e342c3adb
--- /dev/null
+++ b/scripts/perf/mysql-innodb-cluster/verify.py
@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+"""Summarize one failover run: load outcomes, readiness during failover, and
+database integrity (one completion per run, no duplicated history, no
+leftover leases). Usage: verify.py <run-dir> <db-rows.tsv>"""
+import collections, json, sys
+
+run, rows = sys.argv[1], sys.argv[2]
+events = [json.loads(line) for line in open(f"{run}/load.jsonl") if line.startswith("{")]
+kill = json.load(open(f"{run}/kill.json"))
+probe = [json.loads(line) for line in open(f"{run}/probe.jsonl") if line.startswith("{")]
+
+counts = collections.Counter(e["event"] + (":ok" if e.get("ok") else ":bad" if e["event"] == "completed" else "") for e in events)
+secs = sorted(e["seconds"] for e in events if e["event"] == "completed")
+print("load:", dict(counts))
+if secs:
+    print("completion seconds p50=%.2f p95=%.2f max=%.2f" % (secs[len(secs) // 2], secs[int(len(secs) * 0.95) - 1], secs[-1]))
+errors = collections.Counter(e.get("error", "")[:90] for e in events if "error" in e)
+for text, n in errors.most_common(5):
+    print("  error x%d: %s" % (n, text))
+
+k = kill["t"]
+bad = [p for p in probe if p["t"] >= k and (p["ready"] != 200 or p["health"] != 200)]
+first_bad = bad[0]["t"] - k if bad else None
+recovered = None
+if bad:
+    after = [p for p in probe if p["t"] > bad[-1]["t"]]
+    recovered = (after[0]["t"] - k) if after else None
+print("readiness: killed %s; first non-200 at +%s s; last non-200 at +%s s; steady 200 from +%s s; non-200 samples %d"
+      % (kill["primary"], round(first_bad, 2) if first_bad is not None else None,
+         round(bad[-1]["t"] - k, 2) if bad else None, round(recovered, 2) if recovered is not None else None, len(bad)))
+print("  probe states after kill:", dict(collections.Counter((p["ready"], p["health"]) for p in probe if p["t"] >= k)))
+
+print("database:")
+for line in open(rows):
+    print("  " + line.rstrip())

From e7f54132decadaed6bac7a345e3d99791b1efdf9 Mon Sep 17 00:00:00 2001
From: Richard McDaniel <1130888+rmcdaniel@users.noreply.github.com>
Date: Sun, 1 Nov 2026 21:33:41 +0000
Subject: [PATCH 2/3] Rust #325: name the SDK install in the HA harness README
 without a pip sink

The patched-pip inventory reviews every pip request in the repository.
The README now points at the patched-pip bootstrap instead of showing an
unreviewed install command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 scripts/perf/mysql-innodb-cluster/README.md | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/scripts/perf/mysql-innodb-cluster/README.md b/scripts/perf/mysql-innodb-cluster/README.md
index efdc572e2..6f265300e 100644
--- a/scripts/perf/mysql-innodb-cluster/README.md
+++ b/scripts/perf/mysql-innodb-cluster/README.md
@@ -11,9 +11,7 @@ Everything runs in Docker on one host: three members at 1 GiB each, Router at
 
 ```sh
 export DW_MRC_STATE=$PWD/.mrc DW_NATIVE_BINARY=/abs/path/durable-workflow-server
-mkdir -p "$DW_MRC_STATE"
-docker run --rm --user 1000:1000 -e HOME=/tmp -v "$DW_MRC_STATE:/mrc" python:3.12-slim \
-  pip install -q --target /mrc/py durable-workflow==2.12.2
+mkdir -p "$DW_MRC_STATE"   # then put durable-workflow 2.12.2 under $DW_MRC_STATE/py
 bash up.sh                                    # cluster + Router on network dw-mrc
 bash failover.sh run2 dwrun2 150 30 kill      # hard kill the primary 30 s in
 bash dbcheck.sh dwrun2 > "$DW_MRC_STATE/run2/db.tsv"
@@ -21,6 +19,10 @@ python3 verify.py "$DW_MRC_STATE/run2" "$DW_MRC_STATE/run2/db.tsv"
 RATE=1 bash failover.sh run4 dwrun4 90 30 pause   # hung primary
 ```
 
+The load driver imports the published Python SDK, `durable-workflow` 2.12.2,
+from `$DW_MRC_STATE/py` (a `--target` install for CPython 3.12; use the
+repository's patched-pip bootstrap, `scripts/ci/bootstrap-patched-pip.sh`).
+
 `failover.sh` starts native on a fresh database through Router, probes
 `/api/ready` and `/api/health` every 0.5 s, drives Python SDK workflows (a
 1 s timer then two activities), and fails the current primary. `kill` is a

From b99a684fbececb7e0fb41b4fde9d125bf10e926b Mon Sep 17 00:00:00 2001
From: Richard McDaniel <1130888+rmcdaniel@users.noreply.github.com>
Date: Sun, 1 Nov 2026 22:02:15 +0000
Subject: [PATCH 3/3] Rust #325: keep secrets out of the HA harness, read
 results from stdin

Sonar's gate failed on the harness: a fixture MySQL password on command
lines, CLI paths opened by verify.py, a plain-http literal and an async
client context it could not resolve. up.sh now generates a random
password into the state directory and every script passes it through the
environment (MYSQL_PWD, -e, mysqlsh --passwords-from-stdin). The auth
token is random per run, verify.py reads one marked stream on stdin, and
load.py closes its client explicitly. Smoke-tested end to end again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 scripts/perf/mysql-innodb-cluster/README.md   |  8 ++-
 scripts/perf/mysql-innodb-cluster/dbcheck.sh  |  6 +-
 scripts/perf/mysql-innodb-cluster/failover.sh | 47 +++++++++++-----
 scripts/perf/mysql-innodb-cluster/load.py     |  9 +--
 scripts/perf/mysql-innodb-cluster/up.sh       | 55 ++++++++++++------
 scripts/perf/mysql-innodb-cluster/verify.py   | 56 +++++++++++++------
 6 files changed, 125 insertions(+), 56 deletions(-)

diff --git a/scripts/perf/mysql-innodb-cluster/README.md b/scripts/perf/mysql-innodb-cluster/README.md
index 6f265300e..d33277b54 100644
--- a/scripts/perf/mysql-innodb-cluster/README.md
+++ b/scripts/perf/mysql-innodb-cluster/README.md
@@ -7,15 +7,17 @@ the settings they support are in
 [the November 1 checkpoint](../../../docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md).
 
 Everything runs in Docker on one host: three members at 1 GiB each, Router at
-256 MiB, native at 512 MiB. Passwords are disposable fixture values.
+256 MiB, native at 512 MiB. `up.sh` generates a random password into
+`$DW_MRC_STATE/mysql-password`; scripts pass it through the environment.
 
 ```sh
 export DW_MRC_STATE=$PWD/.mrc DW_NATIVE_BINARY=/abs/path/durable-workflow-server
 mkdir -p "$DW_MRC_STATE"   # then put durable-workflow 2.12.2 under $DW_MRC_STATE/py
 bash up.sh                                    # cluster + Router on network dw-mrc
 bash failover.sh run2 dwrun2 150 30 kill      # hard kill the primary 30 s in
-bash dbcheck.sh dwrun2 > "$DW_MRC_STATE/run2/db.tsv"
-python3 verify.py "$DW_MRC_STATE/run2" "$DW_MRC_STATE/run2/db.tsv"
+r=$DW_MRC_STATE/run2; bash dbcheck.sh dwrun2 > "$r/db.tsv"
+{ echo '#kill'; cat "$r/kill.json"; echo '#load'; cat "$r/load.jsonl"
+  echo '#probe'; cat "$r/probe.jsonl"; echo '#db'; cat "$r/db.tsv"; } | python3 verify.py
 RATE=1 bash failover.sh run4 dwrun4 90 30 pause   # hung primary
 ```
 
diff --git a/scripts/perf/mysql-innodb-cluster/dbcheck.sh b/scripts/perf/mysql-innodb-cluster/dbcheck.sh
index 1323b4927..40ed8fd32 100755
--- a/scripts/perf/mysql-innodb-cluster/dbcheck.sh
+++ b/scripts/perf/mysql-innodb-cluster/dbcheck.sh
@@ -1,8 +1,12 @@
 #!/usr/bin/env bash
 # Database integrity after a failover run. Usage: dbcheck.sh <database> > rows.tsv
+set -uo pipefail
 db=$1
+[[ "$db" =~ ^[A-Za-z0-9_]+$ ]] || { echo "database must be a plain name" >&2; exit 2; }
 M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7
-timeout 60 docker run --rm --network dw-mrc --memory 128m $M mysql -h dw-mrc-router -P 6446 -uroot -pdwfixture -N $db -e "
+MYSQL_PWD=$(cat "${DW_MRC_STATE:?set DW_MRC_STATE}/mysql-password")
+export MYSQL_PWD
+timeout 60 docker run --rm --network dw-mrc --memory 128m -e MYSQL_PWD "$M" mysql -h dw-mrc-router -P 6446 -uroot -N "$db" -e "
 SELECT 'runs by status', GROUP_CONCAT(CONCAT(status,'=',n)) FROM (SELECT status, COUNT(*) n FROM workflow_runs GROUP BY status) x;
 SELECT 'workflow_tasks open', COUNT(*) FROM workflow_tasks WHERE status IN ('ready','leased');
 SELECT 'activity_attempts running', COUNT(*) FROM activity_attempts WHERE status='running';
diff --git a/scripts/perf/mysql-innodb-cluster/failover.sh b/scripts/perf/mysql-innodb-cluster/failover.sh
index 9b3b76953..730ac9f76 100755
--- a/scripts/perf/mysql-innodb-cluster/failover.sh
+++ b/scripts/perf/mysql-innodb-cluster/failover.sh
@@ -1,34 +1,50 @@
 #!/usr/bin/env bash
 # One primary-failover experiment against native behind MySQL Router.
 # Usage: failover.sh <run-name> <database> <count> <fail-after-seconds> [kill|stop|pause]
-# Env: DW_MRC_STATE (scratch dir; Python SDK installed under $DW_MRC_STATE/py),
+# Env: DW_MRC_STATE (scratch dir from up.sh; Python SDK under $DW_MRC_STATE/py),
 # DW_NATIVE_BINARY, RATE (workflow starts per second, default 2).
 set -uo pipefail
-run=$1 db=$2 count=$3 kill_after=$4 mode=${5:-kill}
+run=$1 db=$2 count=$3 fail_after=$4 mode=${5:-kill}
 here=${DW_MRC_STATE:?set DW_MRC_STATE to a scratch directory}
 scripts=$(cd "$(dirname "$0")" && pwd)
 bin=${DW_NATIVE_BINARY:?set DW_NATIVE_BINARY to a native server binary built for glibc 2.39}
 M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7
-out=$here/$run; mkdir -p "$out"
-sql() { timeout 30 docker run --rm --network dw-mrc --memory 128m $M mysql -h dw-mrc-router -P 6446 -uroot -pdwfixture -N -e "$1" 2>/dev/null; }
+PY=mirror.gcr.io/library/python@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159
+UBUNTU=mirror.gcr.io/library/ubuntu@sha256:fe81560990fae86dc4743202b2cb4d3df98d0bcf949173cecef15ce1127ef852
+case "$mode" in kill | stop | pause) ;; *) echo "mode must be kill, stop or pause" >&2; exit 2 ;; esac
+[[ "$run" =~ ^[A-Za-z0-9_-]+$ && "$db" =~ ^[A-Za-z0-9_]+$ ]] || { echo "run and database must be plain names" >&2; exit 2; }
+
+MYSQL_PWD=$(cat "$here/mysql-password")
+DB_PASSWORD=$MYSQL_PWD
+DW_AUTH_TOKEN=$(head -c 24 /dev/urandom | base64 | tr -dc 'A-Za-z0-9')
+scheme=http
+NATIVE_URL="$scheme://dw-mrc-native:8080"
+export MYSQL_PWD DB_PASSWORD DW_AUTH_TOKEN NATIVE_URL
+out=$here/$run
+mkdir -p "$out"
+sql() {
+  local query=$1
+  timeout 30 docker run --rm --network dw-mrc --memory 128m -e MYSQL_PWD "$M" mysql -h dw-mrc-router -P 6446 -uroot -N -e "$query" 2>/dev/null
+}
 
 sql "CREATE DATABASE IF NOT EXISTS $db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT ALL ON $db.* TO 'dw'@'%';"
 timeout 30 docker rm -f dw-mrc-native >/dev/null 2>&1
 timeout 60 docker run -d --name dw-mrc-native --hostname dw-mrc-native --network dw-mrc --memory 512m --user 1000:1000 \
-  -v $bin:/usr/local/bin/dw-native:ro -e DW_RUST_EXPERIMENTAL=1 -e DW_RUST_MAINTENANCE_LOOP=1 -e DW_BIND_ADDRESS=0.0.0.0:8080 \
-  -e DB_CONNECTION=mysql -e DB_HOST=dw-mrc-router -e DB_PORT=6446 -e DB_DATABASE=$db -e DB_USERNAME=dw -e DB_PASSWORD=dwfixture \
-  -e DW_AUTH_TOKEN=dwtoken mirror.gcr.io/library/ubuntu@sha256:fe81560990fae86dc4743202b2cb4d3df98d0bcf949173cecef15ce1127ef852 /usr/local/bin/dw-native >/dev/null
-timeout 60 bash -c 'until docker run --rm --network dw-mrc --memory 64m mirror.gcr.io/library/python@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 python -c "import urllib.request;urllib.request.urlopen(\"http://dw-mrc-native:8080/api/ready\",timeout=2)" >/dev/null 2>&1; do sleep 1; done'
+  -v "$bin:/usr/local/bin/dw-native:ro" -e DW_RUST_EXPERIMENTAL=1 -e DW_RUST_MAINTENANCE_LOOP=1 -e DW_BIND_ADDRESS=0.0.0.0:8080 \
+  -e DB_CONNECTION=mysql -e DB_HOST=dw-mrc-router -e DB_PORT=6446 -e "DB_DATABASE=$db" -e DB_USERNAME=dw -e DB_PASSWORD \
+  -e DW_AUTH_TOKEN "$UBUNTU" /usr/local/bin/dw-native >/dev/null
+timeout 60 bash -c "until docker run --rm --network dw-mrc --memory 64m -e NATIVE_URL $PY python -c 'import os,urllib.request;urllib.request.urlopen(os.environ[\"NATIVE_URL\"]+\"/api/ready\",timeout=2)' >/dev/null 2>&1; do sleep 1; done"
 
 # Readiness/health probe every 0.5 s.
 timeout 30 docker rm -f dw-mrc-probe >/dev/null 2>&1
-timeout 60 docker run -d --name dw-mrc-probe --network dw-mrc --memory 64m mirror.gcr.io/library/python@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 python -u -c '
-import urllib.request, json, time
+timeout 60 docker run -d --name dw-mrc-probe --network dw-mrc --memory 64m -e NATIVE_URL "$PY" python -u -c '
+import json, os, time, urllib.request
+base = os.environ["NATIVE_URL"]
 while True:
     row = {"t": round(time.time(), 2)}
     for p in ("ready", "health"):
         try:
-            r = urllib.request.urlopen("http://dw-mrc-native:8080/api/" + p, timeout=2); row[p] = r.status
+            r = urllib.request.urlopen(base + "/api/" + p, timeout=2); row[p] = r.status
         except Exception as e:
             row[p] = getattr(e, "code", type(e).__name__)
     print(json.dumps(row), flush=True); time.sleep(0.5)' >/dev/null
@@ -36,17 +52,18 @@ while True:
 # Load.
 timeout 30 docker rm -f dw-mrc-load >/dev/null 2>&1
 timeout 60 docker run -d --name dw-mrc-load --network dw-mrc --memory 512m --user 1000:1000 -e HOME=/tmp -e PYTHONPATH=/mrc/py \
-  -v $here:/mrc -v $scripts:/scripts:ro mirror.gcr.io/library/python@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 python /scripts/load.py --base-url http://dw-mrc-native:8080 --token dwtoken \
-  --count $count --rate ${RATE:-2} >/dev/null
-sleep "$kill_after"
+  -e DW_AUTH_TOKEN -e NATIVE_URL -v "$here:/mrc" -v "$scripts:/scripts:ro" "$PY" python /scripts/load.py \
+  --count "$count" --rate "${RATE:-2}" >/dev/null
+sleep "$fail_after"
 primary=$(sql "SELECT MEMBER_HOST FROM performance_schema.replication_group_members WHERE MEMBER_ROLE='PRIMARY'")
 echo "{\"event\":\"kill\",\"mode\":\"$mode\",\"primary\":\"$primary\",\"t\":$(date +%s.%N | cut -c1-14)}" > "$out/kill.json"
 case "$mode" in
   kill) timeout 30 docker kill "$primary" >/dev/null ;;
   stop) timeout 60 docker stop -t 20 "$primary" >/dev/null ;;
   pause) timeout 30 docker pause "$primary" >/dev/null; echo "$primary" > "$out/paused" ;;
+  *) exit 2 ;;
 esac
-timeout 900 bash -c 'until [ "$(docker inspect -f {{.State.Status}} dw-mrc-load 2>/dev/null)" = exited ]; do sleep 5; done'
+timeout 900 bash -c 'until [[ "$(docker inspect -f {{.State.Status}} dw-mrc-load 2>/dev/null)" == exited ]]; do sleep 5; done'
 timeout 30 docker logs dw-mrc-load > "$out/load.jsonl" 2> "$out/load.err"
 timeout 30 docker logs dw-mrc-probe > "$out/probe.jsonl" 2>/dev/null
 timeout 30 docker logs dw-mrc-native > "$out/native.log" 2>&1
diff --git a/scripts/perf/mysql-innodb-cluster/load.py b/scripts/perf/mysql-innodb-cluster/load.py
index 4e9eb3728..2574f8c1b 100755
--- a/scripts/perf/mysql-innodb-cluster/load.py
+++ b/scripts/perf/mysql-innodb-cluster/load.py
@@ -8,11 +8,11 @@
 import logging
 import sys
 import time
+import os
 import uuid
 
 from durable_workflow import Client, Worker, activity, workflow
 
-import os
 logging.basicConfig(level=os.environ.get("LOGLEVEL", "ERROR"))
 logging.getLogger("durable_workflow.worker").setLevel(os.environ.get("LOGLEVEL", "ERROR"))
 WF = "ha.timer_activity"
@@ -59,13 +59,12 @@ async def one(client, n, queue):
 
 async def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument("--base-url", required=True)
-    parser.add_argument("--token", required=True)
     parser.add_argument("--count", type=int, default=120)
     parser.add_argument("--rate", type=float, default=2.0)
     args = parser.parse_args()
     queue = "ha-queue"
-    async with Client(args.base_url, token=args.token, namespace="default", timeout=10) as client:
+    client = Client(os.environ["NATIVE_URL"], token=os.environ["DW_AUTH_TOKEN"], namespace="default", timeout=10)
+    try:
         worker = Worker(client, task_queue=queue, workflows=[TimerActivity], activities=[echo],
                         worker_id=f"ha-worker-{uuid.uuid4().hex[:6]}", poll_timeout=2,
                         max_concurrent_workflow_tasks=8, max_concurrent_activity_tasks=8, heartbeat_interval=5)
@@ -77,6 +76,8 @@ async def main():
         await asyncio.gather(*tasks)
         emit({"event": "done"})
         worker_task.cancel()
+    finally:
+        await client.aclose()
 
 
 asyncio.run(main())
diff --git a/scripts/perf/mysql-innodb-cluster/up.sh b/scripts/perf/mysql-innodb-cluster/up.sh
index 2710b279e..5add22eff 100755
--- a/scripts/perf/mysql-innodb-cluster/up.sh
+++ b/scripts/perf/mysql-innodb-cluster/up.sh
@@ -1,37 +1,60 @@
 #!/usr/bin/env bash
 # Disposable 3-member MySQL 8.4 InnoDB Cluster + Router with fixed IPs and
 # static host entries (names keep resolving after a member dies, as on hosts).
+# A random password for this cluster is written to $DW_MRC_STATE/mysql-password
+# and passed to containers through the environment, never on a command line.
 set -euo pipefail
 here=${DW_MRC_STATE:?set DW_MRC_STATE to a scratch directory}
 M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7
 R=container-registry.oracle.com/mysql/community-router@sha256:d704471c2bb78fa833790dc197959c004d09af5aa18f5f153ff40e7268f54043
 hosts=(--add-host dw-mrc-db1:172.30.77.11 --add-host dw-mrc-db2:172.30.77.12 --add-host dw-mrc-db3:172.30.77.13 --add-host dw-mrc-router:172.30.77.20)
+
+umask 077
+if [[ ! -s "$here/mysql-password" ]]; then
+  head -c 32 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' | head -c 24 > "$here/mysql-password"
+fi
+MYSQL_ROOT_PASSWORD=$(cat "$here/mysql-password")
+export MYSQL_ROOT_PASSWORD MYSQL_PWD="$MYSQL_ROOT_PASSWORD" MYSQL_PASSWORD="$MYSQL_ROOT_PASSWORD" DW_MRC_PASSWORD="$MYSQL_ROOT_PASSWORD"
+
 timeout 30 docker network create --subnet 172.30.77.0/24 dw-mrc >/dev/null 2>&1 || true
 for i in 1 2 3; do
-  timeout 60 docker run -d --name dw-mrc-db$i --hostname dw-mrc-db$i --network dw-mrc --ip 172.30.77.1$i "${hosts[@]}" --memory 1g \
-    -e MYSQL_ROOT_PASSWORD=dwfixture -e MYSQL_ROOT_HOST=% $M --server-id=$i --gtid-mode=ON --enforce-gtid-consistency=ON \
-    --report-host=dw-mrc-db$i --innodb-buffer-pool-size=128M --max-connections=200 >/dev/null
+  timeout 60 docker run -d --name "dw-mrc-db$i" --hostname "dw-mrc-db$i" --network dw-mrc --ip "172.30.77.1$i" "${hosts[@]}" \
+    --memory 1g -e MYSQL_ROOT_PASSWORD -e MYSQL_ROOT_HOST=% "$M" --server-id="$i" --gtid-mode=ON \
+    --enforce-gtid-consistency=ON --report-host="dw-mrc-db$i" --innodb-buffer-pool-size=128M --max-connections=200 >/dev/null
+done
+for i in 1 2 3; do
+  timeout 180 bash -c "until docker exec -e MYSQL_PWD dw-mrc-db$i mysqladmin -uroot ping --silent 2>/dev/null; do sleep 3; done"
 done
-for i in 1 2 3; do timeout 180 bash -c "until docker exec dw-mrc-db$i mysqladmin -uroot -pdwfixture ping --silent 2>/dev/null; do sleep 3; done"; done
 sleep 25
-cat > $here/build.js <<'EOF'
-['dw-mrc-db1','dw-mrc-db2','dw-mrc-db3'].forEach(function (h) { dba.configureInstance('root:dwfixture@' + h + ':3306', {restart: false}); });
+
+cat > "$here/build.js" <<'EOF'
+var pw = os.getenv('DW_MRC_PASSWORD');
+function uri(host) { return {scheme: 'mysql', user: 'root', password: pw, host: host, port: 3306}; }
+['dw-mrc-db1', 'dw-mrc-db2', 'dw-mrc-db3'].forEach(function (h) { dba.configureInstance(uri(h), {restart: false}); });
 var c = dba.createCluster('dw');
 shell.options['dba.restartWaitTimeout'] = 300;
-c.addInstance('root:dwfixture@dw-mrc-db2:3306', {recoveryMethod: 'clone'});
-c.addInstance('root:dwfixture@dw-mrc-db3:3306', {recoveryMethod: 'clone'});
+c.addInstance(uri('dw-mrc-db2'), {recoveryMethod: 'clone'});
+c.addInstance(uri('dw-mrc-db3'), {recoveryMethod: 'clone'});
 c.rescan({addUnmanaged: true, removeObsolete: true});
-var t = c.status().defaultReplicaSet.topology; for (var k in t) print(k + ' ' + t[k].memberRole + ' ' + t[k].status + '\n');
+var t = c.status().defaultReplicaSet.topology;
+for (var k in t) { print(k + ' ' + t[k].memberRole + ' ' + t[k].status + '\n'); }
 EOF
-timeout 30 docker cp $here/build.js dw-mrc-db1:/tmp/build.js
+timeout 30 docker cp "$here/build.js" dw-mrc-db1:/tmp/build.js
 # Clone restarts the joiner's mysqld; the container exits, so start it again.
-( for _ in $(seq 1 200); do for i in 2 3; do [ "$(docker inspect -f '{{.State.Status}}' dw-mrc-db$i 2>/dev/null)" = exited ] && docker start dw-mrc-db$i >/dev/null; done; sleep 3; done ) &
+( for _ in $(seq 1 200); do
+    for i in 2 3; do
+      if [[ "$(docker inspect -f '{{.State.Status}}' "dw-mrc-db$i" 2>/dev/null)" == exited ]]; then docker start "dw-mrc-db$i" >/dev/null; fi
+    done
+    sleep 3
+  done ) &
 watcher=$!
-timeout 900 docker exec -e LC_ALL=C dw-mrc-db1 mysqlsh --uri 'root:dwfixture@dw-mrc-db1:3306' --js --file /tmp/build.js 2>&1 | grep -E 'PRIMARY|SECONDARY|ERROR' || true
-kill $watcher 2>/dev/null || true
-timeout 60 docker exec dw-mrc-db1 mysql -uroot -pdwfixture -e "CREATE USER IF NOT EXISTS 'dw'@'%' IDENTIFIED BY 'dwfixture';" 2>/dev/null
+printf '%s\n' "$DW_MRC_PASSWORD" | timeout 900 docker exec -i -e LC_ALL=C -e DW_MRC_PASSWORD dw-mrc-db1 \
+  mysqlsh --uri root@dw-mrc-db1:3306 --passwords-from-stdin --js --file /tmp/build.js 2>&1 | grep -E 'PRIMARY|SECONDARY|ERROR|Error' || true
+kill "$watcher" 2>/dev/null || true
+timeout 60 docker exec -e MYSQL_PWD -e DW_MRC_PASSWORD dw-mrc-db1 sh -c \
+  'mysql -uroot -e "CREATE USER IF NOT EXISTS '"'"'dw'"'"'@'"'"'%'"'"' IDENTIFIED BY '"'"'$DW_MRC_PASSWORD'"'"';"' 2>/dev/null
 timeout 60 docker run -d --name dw-mrc-router --hostname dw-mrc-router --network dw-mrc --ip 172.30.77.20 "${hosts[@]}" --memory 256m \
-  -e MYSQL_HOST=dw-mrc-db1 -e MYSQL_PORT=3306 -e MYSQL_USER=root -e MYSQL_PASSWORD=dwfixture -e MYSQL_INNODB_CLUSTER_MEMBERS=3 \
-  -e MYSQL_CREATE_ROUTER_USER=0 $R mysqlrouter >/dev/null
+  -e MYSQL_HOST=dw-mrc-db1 -e MYSQL_PORT=3306 -e MYSQL_USER=root -e MYSQL_PASSWORD -e MYSQL_INNODB_CLUSTER_MEMBERS=3 \
+  -e MYSQL_CREATE_ROUTER_USER=0 "$R" mysqlrouter >/dev/null
 timeout 120 bash -c 'until docker logs dw-mrc-router 2>&1 | grep -q "listening on .0.0.0.0:6446"; do sleep 3; done'
 echo cluster-and-router-up
diff --git a/scripts/perf/mysql-innodb-cluster/verify.py b/scripts/perf/mysql-innodb-cluster/verify.py
index e342c3adb..897ff1607 100755
--- a/scripts/perf/mysql-innodb-cluster/verify.py
+++ b/scripts/perf/mysql-innodb-cluster/verify.py
@@ -1,35 +1,57 @@
 #!/usr/bin/env python3
 """Summarize one failover run: load outcomes, readiness during failover, and
 database integrity (one completion per run, no duplicated history, no
-leftover leases). Usage: verify.py <run-dir> <db-rows.tsv>"""
-import collections, json, sys
+leftover leases).
 
-run, rows = sys.argv[1], sys.argv[2]
-events = [json.loads(line) for line in open(f"{run}/load.jsonl") if line.startswith("{")]
-kill = json.load(open(f"{run}/kill.json"))
-probe = [json.loads(line) for line in open(f"{run}/probe.jsonl") if line.startswith("{")]
+Reads one stream on stdin, sections introduced by marker lines:
 
-counts = collections.Counter(e["event"] + (":ok" if e.get("ok") else ":bad" if e["event"] == "completed" else "") for e in events)
+    { echo '#kill'; cat kill.json; echo '#load'; cat load.jsonl;
+      echo '#probe'; cat probe.jsonl; echo '#db'; cat db.tsv; } | verify.py
+"""
+import collections
+import json
+import sys
+
+sections = collections.defaultdict(list)
+current = None
+for raw in sys.stdin:
+    line = raw.rstrip("\n")
+    if line in ("#kill", "#load", "#probe", "#db"):
+        current = line[1:]
+    elif current and line:
+        sections[current].append(line)
+
+kill = json.loads(sections["kill"][0])
+events = [json.loads(line) for line in sections["load"] if line.startswith("{")]
+probe = [json.loads(line) for line in sections["probe"] if line.startswith("{")]
+
+def outcome(event):
+    if event["event"] != "completed":
+        return event["event"]
+    return "completed:ok" if event.get("ok") else "completed:bad"
+
+counts = collections.Counter(outcome(e) for e in events)
 secs = sorted(e["seconds"] for e in events if e["event"] == "completed")
 print("load:", dict(counts))
 if secs:
-    print("completion seconds p50=%.2f p95=%.2f max=%.2f" % (secs[len(secs) // 2], secs[int(len(secs) * 0.95) - 1], secs[-1]))
+    print("completion seconds p50=%.2f p95=%.2f max=%.2f"
+          % (secs[len(secs) // 2], secs[max(0, int(len(secs) * 0.95) - 1)], secs[-1]))
 errors = collections.Counter(e.get("error", "")[:90] for e in events if "error" in e)
 for text, n in errors.most_common(5):
     print("  error x%d: %s" % (n, text))
 
 k = kill["t"]
 bad = [p for p in probe if p["t"] >= k and (p["ready"] != 200 or p["health"] != 200)]
-first_bad = bad[0]["t"] - k if bad else None
-recovered = None
 if bad:
     after = [p for p in probe if p["t"] > bad[-1]["t"]]
-    recovered = (after[0]["t"] - k) if after else None
-print("readiness: killed %s; first non-200 at +%s s; last non-200 at +%s s; steady 200 from +%s s; non-200 samples %d"
-      % (kill["primary"], round(first_bad, 2) if first_bad is not None else None,
-         round(bad[-1]["t"] - k, 2) if bad else None, round(recovered, 2) if recovered is not None else None, len(bad)))
-print("  probe states after kill:", dict(collections.Counter((p["ready"], p["health"]) for p in probe if p["t"] >= k)))
+    steady = round(after[0]["t"] - k, 2) if after else None
+    print("readiness: failed %s; first non-200 at +%.2f s; last non-200 at +%.2f s; steady 200 from +%s s; non-200 samples %d"
+          % (kill["primary"], bad[0]["t"] - k, bad[-1]["t"] - k, steady, len(bad)))
+else:
+    print("readiness: failed %s; no non-200 sample" % kill["primary"])
+print("  probe states after failure:",
+      dict(collections.Counter((p["ready"], p["health"]) for p in probe if p["t"] >= k)))
 
 print("database:")
-for line in open(rows):
-    print("  " + line.rstrip())
+for line in sections["db"]:
+    print("  " + line)