Skip to content

Rust #325: qualify native on MySQL InnoDB Cluster behind Router - #928

Merged
rmcdaniel merged 4 commits into
mainfrom
rust-325-mysql-router-ha
Nov 2, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
rust-325-mysql-router-ha

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Qualifies native on a highly available MySQL topology: a three-member single-primary MySQL 8.4 InnoDB Cluster behind MySQL Router 8.4, with native on Router's classic read-write port. Results and operator settings are in docs/rust-server-port-checkpoints/2026-11-01-mysql-innodb-cluster-router.md; the harness is in scripts/perf/mysql-innodb-cluster/.

Results (one disposable host; Python SDK 2.12.2; workflows of a 1 s timer then two activities):

Run Failure Outcome Readiness non-200
Baseline none 60/60, p50 5.1 s none
2 primary crash 106/107 completed; the 107th made ready again at its activity lease expiry +4.4 s to +40.7 s
3 primary clean shutdown 150/150 one sample
4 primary hung (paused) 74/74 +0.5 s to +32.7 s

History is exact in every run: one completion per run, exact event counts, no duplicate (run, sequence) rows, no leftover leases. While no primary is reachable, native answers 503 storage_unavailable, readiness answers 503, and it recovers without a restart, 1 to 5 s after Router switches. Group Replication election and Router's metadata refresh account for the rest.

Defects found and fixed

  • Activity poll without server_time (any database). Published PHP has answered the activity poll with a top-level server_time since 2.11.0. Python SDK 2.12.2 measures a claimed activity's lease against it and refused every activity native delivered (ActivityLeaseAuthorityLost / invalid_activity_claim). Native now adds it to every activity poll answer. Test: activity_poll_answers_carry_the_server_clock.
  • Failover error classes. MySQL 1053, 1290, 1927, 2055, 3100 and 4031 are now backend_unavailable, and 3101 is backend_lock_pressure. Every database error already answered 503; the class drives the bounded diagnostic and the retryable deregistration receipt. Test: mysql_cluster_failover_errors_are_transient.

Checks run locally: Rust lib 384 passed; execution suite on SQLite 667 passed, 1 ignored; node unit suite 6541/6543, with the timing test excluded and the known local-only PhpImageContextTest failure. CI covers MySQL, MariaDB and PostgreSQL.

Not covered: multi-primary, Router read/write split, several Routers, long-running load, network partitions across hosts, TLS to Router, the Rust and PHP SDKs.

🤖 Generated with Claude Code

rmcdaniel and others added 3 commits November 1, 2026 21:23
Native ran on a three-member single-primary MySQL 8.4 InnoDB Cluster
through Router's read-write port: schema creation, the timer and activity
lifecycle, and primary crash, clean shutdown and hang. History stayed
exact with no leftover leases. Native answered 503 while no primary was
reachable and recovered without a restart, 1 to 5 s after Router switched.

Two defects found:
- The activity poll answer lacked PHP's top-level server_time (published
  since 2.11.0). Python SDK 2.12.2 measures a claimed lease against it and
  refused every activity native delivered, on any database.
- MySQL errors a primary switch can raise (1053, 1290, 1927, 2055, 3100,
  4031; 3101 as lock pressure) are now classified transient.

The harness is in scripts/perf/mysql-innodb-cluster; results and operator
settings are in the November 1 checkpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p sink

The patched-pip inventory reviews every pip request in the repository.
The README now points at the patched-pip bootstrap instead of showing an
unreviewed install command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sonar's gate failed on the harness: a fixture MySQL password on command
lines, CLI paths opened by verify.py, a plain-http literal and an async
client context it could not resolve. up.sh now generates a random
password into the state directory and every script passes it through the
environment (MYSQL_PWD, -e, mysqlsh --passwords-from-stdin). The auth
token is random per run, verify.py reads one marked stream on stdin, and
load.py closes its client explicitly. Smoke-tested end to end again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Server Perf run 41355053946: the PHP polling-cache bounded-growth smoke recorded one ready request error, and Performance source qualification failed on that smoke result. The smoke builds and exercises the PHP server image, which this PR does not change; the PR touches native source, docs and a new scripts/perf/ harness. I've rerun only the failed jobs.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Shared Server fixtures run 41355053791: codec-deadline-health recorded outcome: pass for its SQLite cell, then failed the job because cleanup could not confirm that …-codec-deadline-health-sqlite-db had stopped (no such object, so it was already gone). That's a harness cleanup race, not this change. I've rerun only the failed job.

@rmcdaniel

rmcdaniel commented Nov 1, 2026 •

Copy link
Copy Markdown
Member Author

Security exact-head review of b99a684: accepted. The activity-poll top-level server_time restores the existing wire contract for leased, empty, and task-queue-changed answers without changing workflow polls. The added MySQL classifications affect diagnostics and retryable deregistration receipts only and do not broaden authority; backend failures remain 503. The harness uses per-run synthetic credentials, a 0600 state file, pinned images, bounded resources, and no host paths or real secrets. The current codec-deadline-health failure is unrelated to this source: all codec/deadline checks passed, then the Ubuntu 26 Docker client emitted lower-case "error: no such object", hitting the known cleanup-parser defect already fixed in #927. Remaining hosted checks still own ordinary qualification. No Security hold at this exact head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged origin/main (0329bd7, which includes #927's Ubuntu 26 Docker-case fix) into this branch as a merge commit only. There is no force-push and no source change beyond #927's two files (scripts/conformance/server-parity/codec-deadline-cells.sh, tests/Unit/ServerParityCodecDeadlineCellsTest.mjs). The earlier codec-deadline-health failure was the cleanup parser that #927 fixes. Fresh CI is running; I'll enqueue once it's green. (Server, lab2)

@sonarqubecloud

sonarqubecloud Bot commented Nov 2, 2026

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 2, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security review of current exact head 32def4958667d620ea3740ed48ffe66aaa091ef2: clear.

  • Native now returns a top-level microsecond server_time on leased and no-task activity poll answers without changing workflow answers; the test proves a leased expiry is after that clock and covers the empty answer.
  • The added MySQL error classification is confined to transient deregistration receipts. Observed shutdown/read-only/lost-connection/quorum classes map to backend unavailable, certification rollback to lock pressure, while duplicate-key, missing-table, and foreign-key faults remain non-transient.
  • The disposable HA harness validates operator-supplied run/database names, uses pinned images and bounded containers, generates fresh fixture-only credentials under umask 077, passes them by stdin/environment rather than argv, and reads verification data through a marked stdin stream. It adds no host listener, standing credential, or production path.

The current head is an independently reproduced clean merge of reviewed source b99a684f with base 0329bd71, with no manual conflict edits. git diff --check and all current required/security checks pass; the recorded fresh cluster exercise remains the behavioral evidence. No Security hold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant