diff --git a/docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md b/docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md new file mode 100644 index 000000000..b3b768457 --- /dev/null +++ b/docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md @@ -0,0 +1,115 @@ +# Native on MySQL InnoDB Cluster behind MySQL Router (November 1) + +A highly available native deployment on MySQL runs native 3.x on a MySQL 8 +InnoDB Cluster behind MySQL Router. This checkpoint qualifies that topology +on one disposable host with the +[harness](../../scripts/perf/mysql-innodb-cluster/README.md). It is evidence +for the topology and its failure handling, not a capacity measurement. + +## Conditions + +- MySQL 8.4.13 (`mirror.gcr.io/library/mysql@sha256:a0225d1e…`), three + members, single primary, default Group Replication settings (MySQL + communication stack, `group_replication_member_expel_timeout` 5 s), 1 GiB + and a 128 MiB buffer pool each. +- MySQL Router 8.4.10 + (`container-registry.oracle.com/mysql/community-router@sha256:d704471c…`), + bootstrapped defaults; native uses the classic read-write port 6446. +- Native from this branch (main at `bbed2bec` plus the two fixes below), with + `DW_RUST_MAINTENANCE_LOOP=1`, 512 MiB. +- Load: published Python SDK 2.12.2, one worker (eight workflow and eight + activity slots), workflows of a 1 s timer then two activities. + +## Results + +| Run | Failure | Workflows completed | Starts refused (503) | Readiness non-200 | Integrity | +| --- | --- | --- | --- | --- | --- | +| Baseline | none | 60/60, p50 5.1 s | 0 | none | exact | +| 2 | primary crash (`docker kill`) | 106 of 107 started; the 107th made ready again at its activity lease expiry | 43 | +4.4 s to +40.7 s | exact | +| 3 | primary clean shutdown (`docker stop`) | 150/150 | 0 | one sample, ready again by +3.8 s | exact | +| 4 | primary hung (`docker pause`) | 74/74 | 16 | +0.5 s to +32.7 s | exact | + +"Exact" means: every completed run has one `WorkflowCompleted`, one +`TimerFired` and two `ActivityCompleted` events, no `(run, sequence)` history +row is duplicated, and no workflow task or activity attempt is left leased. + +- **Schema creation through Router** works: native created and verified its + schema on a fresh database through port 6446. +- **Crash recovery time** (run 2, measured from the kill): Group Replication + elected the new primary at +23.4 s, Router routed writes to it at +36 s, + native's health answered 200 at +37.0 s and readiness at +40.7 s. Native + adds 1 to 5 s after Router switches; the rest is election and Router's + metadata refresh, which operators can tune. +- **Honest answers during the window**: with no primary, every request, + including starts, answers `503 storage_unavailable`; readiness answers 503 + and health answers 503 or times out at the probe. Nothing answers 200 while + writes cannot commit. Native reconnects without a restart; the scheduler + loop logs `runtime_scheduler_storage_unavailable` once and resumes. +- **In-flight work**: an activity claimed 0.4 s before the crash could not be + completed by its worker during the window. At its lease expiry native + expired the attempt, failed the activity under its retry policy and made a + workflow task ready, so the run resumes; nothing stays leased. +- **Client-visible errors**: the Python SDK's `start_workflow` and + `result()` do not retry 503, so a caller sees `storage_unavailable` for + calls made during the window (19 `result()` calls in run 2, whose runs did + complete server-side). Callers should retry 503 with backoff. +- **Throughput is the driver's limit, not the failover's**: at 2 starts per + second the single SDK worker saturates (a no-failover control had a 73 s + median); run 4 used 1 per second. + +A first crash run used Docker's embedded DNS, which drops a killed +container's name. The surviving members, on the MySQL communication stack, +then reported each other `UNREACHABLE` intermittently and lost quorum; no +primary was elected and the old one could not rejoin. With fixed addresses +and static host entries, as real hosts have, the same crash failed over +cleanly. Cluster members need names that keep resolving while a member is +down. + +## Defects found and fixed + +- **Activity poll without `server_time`** (any topology). Published PHP has + answered `POST /api/worker/activity-tasks/poll` with a top-level + `server_time` since 2.11.0; native did not. Python SDK 2.12.2 measures a + claimed activity's lease against it and refused every activity native + delivered (`ActivityLeaseAuthorityLost`, `invalid_activity_claim`), so no + activity could run under that SDK. Native now adds `server_time` to every + activity poll answer, leased or empty + (`activity_poll_answers_carry_the_server_clock`). +- **Failover error classes**. MySQL errors a primary switch can surface are + now classified transient: `backend_unavailable` for 1053, 1290, 1927, 2055, + 3100 and 4031, `backend_lock_pressure` for 3101 + (`mysql_cluster_failover_errors_are_transient`). Every database error + already answered 503; the class drives the bounded diagnostic and the + retryable worker deregistration receipt. + +## Settings for operators + +- **Connection**: `DB_CONNECTION=mysql`, `DB_HOST=`, `DB_PORT=6446` + (Router's classic read-write port, never the read-only 6447 or the + read/write-split 6450), `DB_DATABASE`, `DB_USERNAME`, `DB_PASSWORD`, plus + `DB_SSLMODE`/`DB_SSLROOTCERT` for TLS to Router. +- **Pool** (fixed in native): four connections per process, 5 s acquire + timeout, connections tested before reuse, `lock_wait_timeout` and + `innodb_lock_wait_timeout` of 5 s per session. Size Router's + `max_connections` for four per native process plus headroom. +- **Probes**: readiness gates traffic. While no primary is reachable, health + and readiness can take up to about 5 s to answer 503 (the pool's acquire + timeout), so give probes a timeout of at least 6 s. Do not restart native on + a database outage: health is 503 for the whole failover window and a + restart does not shorten it. Expect 35 to 45 s of 503 after a primary crash + with default Group Replication and Router settings, and a few seconds after + a clean shutdown. +- **Router**: keep `connection_sharing` off (the default); native sets + session variables on connect. Router's metadata `ttl` and Group Replication + notifications, and Group Replication's `member_expel_timeout`, set the + failover window. +- **Named locks**: native takes MySQL `GET_LOCK` only while bootstrapping or + upgrading its schema and in upgrade takeover, never at runtime. Named locks + are local to the member, so a failover during a schema upgrade releases the + lock; run upgrades with one native process. + +## Not covered + +Multi-primary mode, Router read/write split, Router on a separate host or +several Routers, long-running load, cross-host network partitions, TLS to +Router, and the Rust and PHP SDKs. diff --git a/docs/rust-server-port.md b/docs/rust-server-port.md index 85cd9aefd..d64b1a804 100644 --- a/docs/rust-server-port.md +++ b/docs/rust-server-port.md @@ -272,6 +272,18 @@ These are confirmed differences from the published PHP reference and open contract decisions found while qualifying native slices. Each one needs a contract fixture before its correction. +- **MySQL InnoDB Cluster behind MySQL Router** (November 1, qualified on one + host; [checkpoint](rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md)). + Native runs on a single-primary InnoDB Cluster through Router's + read-write port. Schema creation, the workflow, timer and + activity lifecycle, and primary crash, clean shutdown and hang all pass + with exact history and no leftover leases; native answers 503 while no + primary is reachable and recovers without a restart, 1 to 5 s after Router + switches. Two defects found and fixed: the activity poll answer lacked + PHP's `server_time` (since 2.11.0), so Python SDK 2.12.2 refused every + activity native delivered on any database; and MySQL failover errors + (1053, 1290, 1927, 2055, 3100, 3101, 4031) are now classified transient. + Multi-primary, read/write split and several Routers are not covered. - **History export waits and timeline** (November 1, closed, [checkpoint](rust-server-port-checkpoints/2026-11-01-history-export-waits.md)). Native's export now serves the run's stored waits with each kind's diff --git a/rust/src/runtime/store.rs b/rust/src/runtime/store.rs index f58159032..e4ce6d83e 100644 --- a/rust/src/runtime/store.rs +++ b/rust/src/runtime/store.rs @@ -266,6 +266,16 @@ fn after(seconds: i64) -> DateTime { now() + chrono::Duration::seconds(seconds) } +/// As PHP's `ActivityTaskController::poll` (since 2.11.0): an activity poll +/// answer carries `server_time`, the clock its lease expiry is measured +/// against. Published SDKs refuse a claimed activity without it. +fn activity_server_time(kind: &str, mut answer: Value) -> Value { + if kind == "activity" { + answer["server_time"] = json!(super::activity_lease_renewal::micros(now())); + } + answer +} + impl Store where for<'c> &'c mut DB::Connection: Executor<'c, Database = DB>, @@ -1974,19 +1984,22 @@ where // Compressed after the claim commits; a replayed receipt keeps // the plain events and takes this request's encoding. super::history_pages::compress(&mut task, encoding)?; - return Ok( + return Ok(activity_server_time( + kind, json!({"task": task, "poll_status": "leased", "server_capabilities": capabilities()}), - ); + )); } if tokio::time::Instant::now() >= deadline { - return self.no_task_answer(&body, poll_request.as_ref(), "empty", encoding).await; + return self.no_task_answer(&body, poll_request.as_ref(), "empty", encoding).await + .map(|answer| activity_server_time(kind, answer)); } if self .poll_queue_changed(&body, kind, &mut cross_kind) .await? && tokio::time::Instant::now() < deadline { - return self.no_task_answer(&body, poll_request.as_ref(), "task_queue_changed", encoding).await; + return self.no_task_answer(&body, poll_request.as_ref(), "task_queue_changed", encoding).await + .map(|answer| activity_server_time(kind, answer)); } #[cfg(debug_assertions)] if let Some(request) = &poll_request { diff --git a/rust/src/runtime/worker_incarnations.rs b/rust/src/runtime/worker_incarnations.rs index 6335e3737..2100717f3 100644 --- a/rust/src/runtime/worker_incarnations.rs +++ b/rust/src/runtime/worker_incarnations.rs @@ -23,11 +23,7 @@ pub(super) fn transient_reason(error: &RuntimeError) -> Option<&'static str> { ) => Some("backend_unavailable"), RuntimeError::Database(sqlx::Error::Database(error)) => { if let Some(mysql) = error.try_downcast_ref::() { - return match mysql.number() { - 1205 | 1213 => Some("backend_lock_pressure"), - 2002 | 2003 | 2006 | 2013 => Some("backend_unavailable"), - _ => None, - }; + return mysql_transient_reason(mysql.number()); } let code = error.code()?; if matches!(code.as_ref(), "55P03" | "40P01" | "40001") @@ -50,6 +46,35 @@ pub(super) fn transient_reason(error: &RuntimeError) -> Option<&'static str> { } } +/// MySQL server and client error numbers that describe a passing backend +/// condition rather than a request fault. Beyond lock waits and lost +/// connections, an InnoDB Cluster primary switch behind MySQL Router can +/// surface: 1053 server shutdown in progress, 1290 a write reaching a member +/// that is (now) `super_read_only`, 1927 connection killed, 2055 connection +/// lost, 3100 a Group Replication commit hook refusal while the member has no +/// quorum, 4031 a server-side disconnect; and 3101 a transaction rolled back +/// at commit by group certification, which a retry can pass. +pub(super) fn mysql_transient_reason(number: u16) -> Option<&'static str> { + match number { + 1205 | 1213 | 3101 => Some("backend_lock_pressure"), + 1053 | 1290 | 1927 | 2002 | 2003 | 2006 | 2013 | 2055 | 3100 | 4031 => Some("backend_unavailable"), + _ => None, + } +} + +#[test] +fn mysql_cluster_failover_errors_are_transient() { + for number in [1053, 1290, 1927, 2002, 2003, 2006, 2013, 2055, 3100, 4031] { + assert_eq!(mysql_transient_reason(number), Some("backend_unavailable"), "{number}"); + } + for number in [1205, 1213, 3101] { + assert_eq!(mysql_transient_reason(number), Some("backend_lock_pressure"), "{number}"); + } + for number in [1062, 1146, 1452] { + assert_eq!(mysql_transient_reason(number), None, "{number}"); + } +} + fn failure(status: StatusCode, reason: &'static str, worker_id: &str, token: &str) -> RuntimeError { RuntimeError::Protocol { status, diff --git a/rust/tests/execution/activity_results.rs b/rust/tests/execution/activity_results.rs index d8b2657ba..2c9270943 100644 --- a/rust/tests/execution/activity_results.rs +++ b/rust/tests/execution/activity_results.rs @@ -354,3 +354,48 @@ async fn activity_result_read_uses_canonical_retry_failure_and_cancellation_outc runtime.close().await; database.remove().await; } + +/// PHP's activity poll answer carries `server_time` (since 2.11.0), leased +/// or empty. Published SDKs measure the lease against it and refuse a claim +/// without it: Python SDK 2.12.2 raised `ActivityLeaseAuthorityLost` +/// (`invalid_activity_claim`) for every activity native delivered. +#[tokio::test] +async fn activity_poll_answers_carry_the_server_clock() { + let database = TestDatabase::new().await; + let runtime = database.open().await.unwrap(); + let app = router(runtime.clone()); + let micros = |value: &Value| { + let text = value.as_str().unwrap_or_default(); + chrono::DateTime::parse_from_rfc3339(text).is_ok() + && text.ends_with('Z') + && text.rsplit_once('.').is_some_and(|(_, fraction)| fraction.len() == 7) + }; + let app = &app; + let call = |method, path: String, body| async move { + namespaces::scoped(app, "default", method, &path, body).await + }; + let poll = || call("POST", "/api/worker/activity-tasks/poll".into(), + json!({"worker_id":"clock-worker","task_queue":"test","timeout_seconds":0})); + let registered = call("POST", "/api/worker/register".into(), json!({ + "worker_id":"clock-worker","task_queue":"test","runtime":"python", + "supported_workflow_types":["echo"],"supported_activity_types":["clock-activity"]})).await; + assert_eq!(registered.0, StatusCode::CREATED, "{}", registered.1); + namespaces::start_scoped(&app, "default", "clock-run").await; + let workflow = call("POST", "/api/worker/workflow-tasks/poll".into(), + json!({"worker_id":"clock-worker","task_queue":"test","timeout_seconds":0})).await.1["task"].clone(); + let done = call("POST", format!("/api/worker/workflow-tasks/{}/complete", workflow["task_id"].as_str().unwrap()), + completion(&workflow, json!([{"type":"schedule_activity","activity_type":"clock-activity", + "arguments":envelope(Payload::Array(vec![])),"start_to_close_timeout":120}]))).await; + assert_eq!(done.0, StatusCode::OK, "{}", done.1); + let leased = poll().await; + assert_eq!(leased.0, StatusCode::OK, "{}", leased.1); + assert_eq!(leased.1["poll_status"], "leased", "{}", leased.1); + assert!(micros(&leased.1["server_time"]), "{}", leased.1); + let expiry = chrono::DateTime::parse_from_rfc3339(leased.1["task"]["lease_expires_at"].as_str().unwrap()).unwrap(); + let clock = chrono::DateTime::parse_from_rfc3339(leased.1["server_time"].as_str().unwrap()).unwrap(); + assert!(expiry > clock, "{}", leased.1); + let empty = poll().await; + assert_eq!(empty.0, StatusCode::OK, "{}", empty.1); + assert_eq!(empty.1["poll_status"], "empty", "{}", empty.1); + assert!(micros(&empty.1["server_time"]), "{}", empty.1); +} diff --git a/scripts/perf/mysql-innodb-cluster/README.md b/scripts/perf/mysql-innodb-cluster/README.md new file mode 100644 index 000000000..d33277b54 --- /dev/null +++ b/scripts/perf/mysql-innodb-cluster/README.md @@ -0,0 +1,42 @@ +# Native behind MySQL InnoDB Cluster and MySQL Router + +A disposable harness that qualifies the native server on a highly available +MySQL topology: a three-member single-primary MySQL 8.4 InnoDB Cluster behind MySQL +Router, with native connected to Router's classic read-write port. Results and +the settings they support are in +[the November 1 checkpoint](../../../docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md). + +Everything runs in Docker on one host: three members at 1 GiB each, Router at +256 MiB, native at 512 MiB. `up.sh` generates a random password into +`$DW_MRC_STATE/mysql-password`; scripts pass it through the environment. + +```sh +export DW_MRC_STATE=$PWD/.mrc DW_NATIVE_BINARY=/abs/path/durable-workflow-server +mkdir -p "$DW_MRC_STATE" # then put durable-workflow 2.12.2 under $DW_MRC_STATE/py +bash up.sh # cluster + Router on network dw-mrc +bash failover.sh run2 dwrun2 150 30 kill # hard kill the primary 30 s in +r=$DW_MRC_STATE/run2; bash dbcheck.sh dwrun2 > "$r/db.tsv" +{ echo '#kill'; cat "$r/kill.json"; echo '#load'; cat "$r/load.jsonl" + echo '#probe'; cat "$r/probe.jsonl"; echo '#db'; cat "$r/db.tsv"; } | python3 verify.py +RATE=1 bash failover.sh run4 dwrun4 90 30 pause # hung primary +``` + +The load driver imports the published Python SDK, `durable-workflow` 2.12.2, +from `$DW_MRC_STATE/py` (a `--target` install for CPython 3.12; use the +repository's patched-pip bootstrap, `scripts/ci/bootstrap-patched-pip.sh`). + +`failover.sh` starts native on a fresh database through Router, probes +`/api/ready` and `/api/health` every 0.5 s, drives Python SDK workflows (a +1 s timer then two activities), and fails the current primary. `kill` is a +crash, `stop` a clean shutdown, `pause` a hung host. Restart or unpause the +member afterwards; it rejoins on its own. `verify.py` reports outcomes, +readiness through the failover, and database integrity: one completion per +run, exact history counts, no duplicate history rows, no leftover leases. + +Members get fixed addresses and static host entries, so a dead member's name +still resolves, as it does on real hosts. Docker's embedded DNS drops a killed +container's name, and the members then lost quorum entirely (see the +checkpoint). + +Tear down: `docker rm -f -v dw-mrc-native dw-mrc-router dw-mrc-db1 dw-mrc-db2 +dw-mrc-db3 && docker network rm dw-mrc`. diff --git a/scripts/perf/mysql-innodb-cluster/dbcheck.sh b/scripts/perf/mysql-innodb-cluster/dbcheck.sh new file mode 100755 index 000000000..40ed8fd32 --- /dev/null +++ b/scripts/perf/mysql-innodb-cluster/dbcheck.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Database integrity after a failover run. Usage: dbcheck.sh > rows.tsv +set -uo pipefail +db=$1 +[[ "$db" =~ ^[A-Za-z0-9_]+$ ]] || { echo "database must be a plain name" >&2; exit 2; } +M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7 +MYSQL_PWD=$(cat "${DW_MRC_STATE:?set DW_MRC_STATE}/mysql-password") +export MYSQL_PWD +timeout 60 docker run --rm --network dw-mrc --memory 128m -e MYSQL_PWD "$M" mysql -h dw-mrc-router -P 6446 -uroot -N "$db" -e " +SELECT 'runs by status', GROUP_CONCAT(CONCAT(status,'=',n)) FROM (SELECT status, COUNT(*) n FROM workflow_runs GROUP BY status) x; +SELECT 'workflow_tasks open', COUNT(*) FROM workflow_tasks WHERE status IN ('ready','leased'); +SELECT 'activity_attempts running', COUNT(*) FROM activity_attempts WHERE status='running'; +SELECT 'runs with !=1 WorkflowCompleted (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='WorkflowCompleted')<>1; +SELECT 'runs with !=2 ActivityCompleted (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='ActivityCompleted')<>2; +SELECT 'runs with !=1 TimerFired (completed runs)', COUNT(*) FROM workflow_runs r WHERE r.status='completed' AND (SELECT COUNT(*) FROM workflow_history_events e WHERE e.workflow_run_id=r.id AND e.event_type='TimerFired')<>1; +SELECT 'duplicate (run,sequence) history rows', COUNT(*) FROM (SELECT workflow_run_id, sequence FROM workflow_history_events GROUP BY 1,2 HAVING COUNT(*)>1) d; +SELECT 'history events by type', GROUP_CONCAT(CONCAT(event_type,'=',n) ORDER BY event_type) FROM (SELECT event_type, COUNT(*) n FROM workflow_history_events GROUP BY 1) y; +" 2>/dev/null | sed 's/\t/: /' diff --git a/scripts/perf/mysql-innodb-cluster/failover.sh b/scripts/perf/mysql-innodb-cluster/failover.sh new file mode 100755 index 000000000..730ac9f76 --- /dev/null +++ b/scripts/perf/mysql-innodb-cluster/failover.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# One primary-failover experiment against native behind MySQL Router. +# Usage: failover.sh [kill|stop|pause] +# Env: DW_MRC_STATE (scratch dir from up.sh; Python SDK under $DW_MRC_STATE/py), +# DW_NATIVE_BINARY, RATE (workflow starts per second, default 2). +set -uo pipefail +run=$1 db=$2 count=$3 fail_after=$4 mode=${5:-kill} +here=${DW_MRC_STATE:?set DW_MRC_STATE to a scratch directory} +scripts=$(cd "$(dirname "$0")" && pwd) +bin=${DW_NATIVE_BINARY:?set DW_NATIVE_BINARY to a native server binary built for glibc 2.39} +M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7 +PY=mirror.gcr.io/library/python@sha256:aa943ce411423c7582288f18707ff66eddcbaeb510502d747a72aeb34f30d159 +UBUNTU=mirror.gcr.io/library/ubuntu@sha256:fe81560990fae86dc4743202b2cb4d3df98d0bcf949173cecef15ce1127ef852 +case "$mode" in kill | stop | pause) ;; *) echo "mode must be kill, stop or pause" >&2; exit 2 ;; esac +[[ "$run" =~ ^[A-Za-z0-9_-]+$ && "$db" =~ ^[A-Za-z0-9_]+$ ]] || { echo "run and database must be plain names" >&2; exit 2; } + +MYSQL_PWD=$(cat "$here/mysql-password") +DB_PASSWORD=$MYSQL_PWD +DW_AUTH_TOKEN=$(head -c 24 /dev/urandom | base64 | tr -dc 'A-Za-z0-9') +scheme=http +NATIVE_URL="$scheme://dw-mrc-native:8080" +export MYSQL_PWD DB_PASSWORD DW_AUTH_TOKEN NATIVE_URL +out=$here/$run +mkdir -p "$out" +sql() { + local query=$1 + timeout 30 docker run --rm --network dw-mrc --memory 128m -e MYSQL_PWD "$M" mysql -h dw-mrc-router -P 6446 -uroot -N -e "$query" 2>/dev/null +} + +sql "CREATE DATABASE IF NOT EXISTS $db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT ALL ON $db.* TO 'dw'@'%';" +timeout 30 docker rm -f dw-mrc-native >/dev/null 2>&1 +timeout 60 docker run -d --name dw-mrc-native --hostname dw-mrc-native --network dw-mrc --memory 512m --user 1000:1000 \ + -v "$bin:/usr/local/bin/dw-native:ro" -e DW_RUST_EXPERIMENTAL=1 -e DW_RUST_MAINTENANCE_LOOP=1 -e DW_BIND_ADDRESS=0.0.0.0:8080 \ + -e DB_CONNECTION=mysql -e DB_HOST=dw-mrc-router -e DB_PORT=6446 -e "DB_DATABASE=$db" -e DB_USERNAME=dw -e DB_PASSWORD \ + -e DW_AUTH_TOKEN "$UBUNTU" /usr/local/bin/dw-native >/dev/null +timeout 60 bash -c "until docker run --rm --network dw-mrc --memory 64m -e NATIVE_URL $PY python -c 'import os,urllib.request;urllib.request.urlopen(os.environ[\"NATIVE_URL\"]+\"/api/ready\",timeout=2)' >/dev/null 2>&1; do sleep 1; done" + +# Readiness/health probe every 0.5 s. +timeout 30 docker rm -f dw-mrc-probe >/dev/null 2>&1 +timeout 60 docker run -d --name dw-mrc-probe --network dw-mrc --memory 64m -e NATIVE_URL "$PY" python -u -c ' +import json, os, time, urllib.request +base = os.environ["NATIVE_URL"] +while True: + row = {"t": round(time.time(), 2)} + for p in ("ready", "health"): + try: + r = urllib.request.urlopen(base + "/api/" + p, timeout=2); row[p] = r.status + except Exception as e: + row[p] = getattr(e, "code", type(e).__name__) + print(json.dumps(row), flush=True); time.sleep(0.5)' >/dev/null + +# Load. +timeout 30 docker rm -f dw-mrc-load >/dev/null 2>&1 +timeout 60 docker run -d --name dw-mrc-load --network dw-mrc --memory 512m --user 1000:1000 -e HOME=/tmp -e PYTHONPATH=/mrc/py \ + -e DW_AUTH_TOKEN -e NATIVE_URL -v "$here:/mrc" -v "$scripts:/scripts:ro" "$PY" python /scripts/load.py \ + --count "$count" --rate "${RATE:-2}" >/dev/null +sleep "$fail_after" +primary=$(sql "SELECT MEMBER_HOST FROM performance_schema.replication_group_members WHERE MEMBER_ROLE='PRIMARY'") +echo "{\"event\":\"kill\",\"mode\":\"$mode\",\"primary\":\"$primary\",\"t\":$(date +%s.%N | cut -c1-14)}" > "$out/kill.json" +case "$mode" in + kill) timeout 30 docker kill "$primary" >/dev/null ;; + stop) timeout 60 docker stop -t 20 "$primary" >/dev/null ;; + pause) timeout 30 docker pause "$primary" >/dev/null; echo "$primary" > "$out/paused" ;; + *) exit 2 ;; +esac +timeout 900 bash -c 'until [[ "$(docker inspect -f {{.State.Status}} dw-mrc-load 2>/dev/null)" == exited ]]; do sleep 5; done' +timeout 30 docker logs dw-mrc-load > "$out/load.jsonl" 2> "$out/load.err" +timeout 30 docker logs dw-mrc-probe > "$out/probe.jsonl" 2>/dev/null +timeout 30 docker logs dw-mrc-native > "$out/native.log" 2>&1 +timeout 30 docker rm -f dw-mrc-probe dw-mrc-load >/dev/null +echo "$primary" > "$out/killed-primary" diff --git a/scripts/perf/mysql-innodb-cluster/load.py b/scripts/perf/mysql-innodb-cluster/load.py new file mode 100755 index 000000000..2574f8c1b --- /dev/null +++ b/scripts/perf/mysql-innodb-cluster/load.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +"""HA load for native behind MySQL Router: start workflows continuously, +wait for each result, and record outcomes and timing as JSON lines.""" + +import argparse +import asyncio +import json +import logging +import sys +import time +import os +import uuid + +from durable_workflow import Client, Worker, activity, workflow + +logging.basicConfig(level=os.environ.get("LOGLEVEL", "ERROR")) +logging.getLogger("durable_workflow.worker").setLevel(os.environ.get("LOGLEVEL", "ERROR")) +WF = "ha.timer_activity" +ACT = "ha.echo" + + +@activity.defn(name=ACT) +async def echo(value): + await asyncio.sleep(0.05) + return value + + +@workflow.defn(name=WF) +class TimerActivity: + def run(self, context, request): + yield context.sleep(1) + first = yield context.schedule_activity(ACT, [{"step": 1, "n": request["n"]}]) + second = yield context.schedule_activity(ACT, [{"step": 2, "n": request["n"]}]) + return {"n": request["n"], "first": first, "second": second} + + +def emit(record): + record["t"] = round(time.time(), 3) + sys.stdout.write(json.dumps(record) + "\n") + sys.stdout.flush() + + +async def one(client, n, queue): + started = time.monotonic() + wid = f"ha-{n}-{uuid.uuid4().hex[:8]}" + try: + handle = await client.start_workflow(workflow_type=WF, task_queue=queue, workflow_id=wid, input=[{"n": n}]) + except Exception as error: # start refused or connection lost + emit({"event": "start_error", "n": n, "wid": wid, "error": f"{type(error).__name__}: {str(error)[:200]}"}) + return + emit({"event": "started", "n": n, "wid": wid}) + try: + result = await handle.result(timeout=300, poll_interval=0.5) + ok = result == {"n": n, "first": {"step": 1, "n": n}, "second": {"step": 2, "n": n}} + emit({"event": "completed", "n": n, "wid": wid, "ok": ok, "seconds": round(time.monotonic() - started, 3)}) + except Exception as error: + emit({"event": "result_error", "n": n, "wid": wid, "error": f"{type(error).__name__}: {str(error)[:200]}"}) + + +async def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--count", type=int, default=120) + parser.add_argument("--rate", type=float, default=2.0) + args = parser.parse_args() + queue = "ha-queue" + client = Client(os.environ["NATIVE_URL"], token=os.environ["DW_AUTH_TOKEN"], namespace="default", timeout=10) + try: + worker = Worker(client, task_queue=queue, workflows=[TimerActivity], activities=[echo], + worker_id=f"ha-worker-{uuid.uuid4().hex[:6]}", poll_timeout=2, + max_concurrent_workflow_tasks=8, max_concurrent_activity_tasks=8, heartbeat_interval=5) + worker_task = asyncio.create_task(worker.run()) + tasks = [] + for n in range(args.count): + tasks.append(asyncio.create_task(one(client, n, queue))) + await asyncio.sleep(1.0 / args.rate) + await asyncio.gather(*tasks) + emit({"event": "done"}) + worker_task.cancel() + finally: + await client.aclose() + + +asyncio.run(main()) diff --git a/scripts/perf/mysql-innodb-cluster/up.sh b/scripts/perf/mysql-innodb-cluster/up.sh new file mode 100755 index 000000000..5add22eff --- /dev/null +++ b/scripts/perf/mysql-innodb-cluster/up.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Disposable 3-member MySQL 8.4 InnoDB Cluster + Router with fixed IPs and +# static host entries (names keep resolving after a member dies, as on hosts). +# A random password for this cluster is written to $DW_MRC_STATE/mysql-password +# and passed to containers through the environment, never on a command line. +set -euo pipefail +here=${DW_MRC_STATE:?set DW_MRC_STATE to a scratch directory} +M=mirror.gcr.io/library/mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7 +R=container-registry.oracle.com/mysql/community-router@sha256:d704471c2bb78fa833790dc197959c004d09af5aa18f5f153ff40e7268f54043 +hosts=(--add-host dw-mrc-db1:172.30.77.11 --add-host dw-mrc-db2:172.30.77.12 --add-host dw-mrc-db3:172.30.77.13 --add-host dw-mrc-router:172.30.77.20) + +umask 077 +if [[ ! -s "$here/mysql-password" ]]; then + head -c 32 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' | head -c 24 > "$here/mysql-password" +fi +MYSQL_ROOT_PASSWORD=$(cat "$here/mysql-password") +export MYSQL_ROOT_PASSWORD MYSQL_PWD="$MYSQL_ROOT_PASSWORD" MYSQL_PASSWORD="$MYSQL_ROOT_PASSWORD" DW_MRC_PASSWORD="$MYSQL_ROOT_PASSWORD" + +timeout 30 docker network create --subnet 172.30.77.0/24 dw-mrc >/dev/null 2>&1 || true +for i in 1 2 3; do + timeout 60 docker run -d --name "dw-mrc-db$i" --hostname "dw-mrc-db$i" --network dw-mrc --ip "172.30.77.1$i" "${hosts[@]}" \ + --memory 1g -e MYSQL_ROOT_PASSWORD -e MYSQL_ROOT_HOST=% "$M" --server-id="$i" --gtid-mode=ON \ + --enforce-gtid-consistency=ON --report-host="dw-mrc-db$i" --innodb-buffer-pool-size=128M --max-connections=200 >/dev/null +done +for i in 1 2 3; do + timeout 180 bash -c "until docker exec -e MYSQL_PWD dw-mrc-db$i mysqladmin -uroot ping --silent 2>/dev/null; do sleep 3; done" +done +sleep 25 + +cat > "$here/build.js" <<'EOF' +var pw = os.getenv('DW_MRC_PASSWORD'); +function uri(host) { return {scheme: 'mysql', user: 'root', password: pw, host: host, port: 3306}; } +['dw-mrc-db1', 'dw-mrc-db2', 'dw-mrc-db3'].forEach(function (h) { dba.configureInstance(uri(h), {restart: false}); }); +var c = dba.createCluster('dw'); +shell.options['dba.restartWaitTimeout'] = 300; +c.addInstance(uri('dw-mrc-db2'), {recoveryMethod: 'clone'}); +c.addInstance(uri('dw-mrc-db3'), {recoveryMethod: 'clone'}); +c.rescan({addUnmanaged: true, removeObsolete: true}); +var t = c.status().defaultReplicaSet.topology; +for (var k in t) { print(k + ' ' + t[k].memberRole + ' ' + t[k].status + '\n'); } +EOF +timeout 30 docker cp "$here/build.js" dw-mrc-db1:/tmp/build.js +# Clone restarts the joiner's mysqld; the container exits, so start it again. +( for _ in $(seq 1 200); do + for i in 2 3; do + if [[ "$(docker inspect -f '{{.State.Status}}' "dw-mrc-db$i" 2>/dev/null)" == exited ]]; then docker start "dw-mrc-db$i" >/dev/null; fi + done + sleep 3 + done ) & +watcher=$! +printf '%s\n' "$DW_MRC_PASSWORD" | timeout 900 docker exec -i -e LC_ALL=C -e DW_MRC_PASSWORD dw-mrc-db1 \ + mysqlsh --uri root@dw-mrc-db1:3306 --passwords-from-stdin --js --file /tmp/build.js 2>&1 | grep -E 'PRIMARY|SECONDARY|ERROR|Error' || true +kill "$watcher" 2>/dev/null || true +timeout 60 docker exec -e MYSQL_PWD -e DW_MRC_PASSWORD dw-mrc-db1 sh -c \ + 'mysql -uroot -e "CREATE USER IF NOT EXISTS '"'"'dw'"'"'@'"'"'%'"'"' IDENTIFIED BY '"'"'$DW_MRC_PASSWORD'"'"';"' 2>/dev/null +timeout 60 docker run -d --name dw-mrc-router --hostname dw-mrc-router --network dw-mrc --ip 172.30.77.20 "${hosts[@]}" --memory 256m \ + -e MYSQL_HOST=dw-mrc-db1 -e MYSQL_PORT=3306 -e MYSQL_USER=root -e MYSQL_PASSWORD -e MYSQL_INNODB_CLUSTER_MEMBERS=3 \ + -e MYSQL_CREATE_ROUTER_USER=0 "$R" mysqlrouter >/dev/null +timeout 120 bash -c 'until docker logs dw-mrc-router 2>&1 | grep -q "listening on .0.0.0.0:6446"; do sleep 3; done' +echo cluster-and-router-up diff --git a/scripts/perf/mysql-innodb-cluster/verify.py b/scripts/perf/mysql-innodb-cluster/verify.py new file mode 100755 index 000000000..897ff1607 --- /dev/null +++ b/scripts/perf/mysql-innodb-cluster/verify.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 +"""Summarize one failover run: load outcomes, readiness during failover, and +database integrity (one completion per run, no duplicated history, no +leftover leases). + +Reads one stream on stdin, sections introduced by marker lines: + + { echo '#kill'; cat kill.json; echo '#load'; cat load.jsonl; + echo '#probe'; cat probe.jsonl; echo '#db'; cat db.tsv; } | verify.py +""" +import collections +import json +import sys + +sections = collections.defaultdict(list) +current = None +for raw in sys.stdin: + line = raw.rstrip("\n") + if line in ("#kill", "#load", "#probe", "#db"): + current = line[1:] + elif current and line: + sections[current].append(line) + +kill = json.loads(sections["kill"][0]) +events = [json.loads(line) for line in sections["load"] if line.startswith("{")] +probe = [json.loads(line) for line in sections["probe"] if line.startswith("{")] + +def outcome(event): + if event["event"] != "completed": + return event["event"] + return "completed:ok" if event.get("ok") else "completed:bad" + +counts = collections.Counter(outcome(e) for e in events) +secs = sorted(e["seconds"] for e in events if e["event"] == "completed") +print("load:", dict(counts)) +if secs: + print("completion seconds p50=%.2f p95=%.2f max=%.2f" + % (secs[len(secs) // 2], secs[max(0, int(len(secs) * 0.95) - 1)], secs[-1])) +errors = collections.Counter(e.get("error", "")[:90] for e in events if "error" in e) +for text, n in errors.most_common(5): + print(" error x%d: %s" % (n, text)) + +k = kill["t"] +bad = [p for p in probe if p["t"] >= k and (p["ready"] != 200 or p["health"] != 200)] +if bad: + after = [p for p in probe if p["t"] > bad[-1]["t"]] + steady = round(after[0]["t"] - k, 2) if after else None + print("readiness: failed %s; first non-200 at +%.2f s; last non-200 at +%.2f s; steady 200 from +%s s; non-200 samples %d" + % (kill["primary"], bad[0]["t"] - k, bad[-1]["t"] - k, steady, len(bad))) +else: + print("readiness: failed %s; no non-200 sample" % kill["primary"]) +print(" probe states after failure:", + dict(collections.Counter((p["ready"], p["health"]) for p in probe if p["t"] >= k))) + +print("database:") +for line in sections["db"]: + print(" " + line)