Add MySQL group replication auto-configuration module by TonyLi15 · Pull Request #94 · Tatzhiro/LineairDB-storage-engine

TonyLi15 · 2025-12-23T08:43:28Z

Adds repl/ module for automatic MySQL Group Replication setup with LineairDB
Provides shell scripts and Python API to manage cluster lifecycle (init, start, stop, cleanup)
Configures local MySQL as primary with Docker-based secondaries
Installs LineairDB plugin on all nodes automatically

Tatzhiro

The README is very difficult to read and understand.
Please explain more in detail about what the user would want to use.
My guess is the user only needs to execute run_all.sh to set up replication, but I am not sure.
Explain like, if you want to set up group replication just run XXX, and if you want to benchmark just run YYY.

Also, what does the benchmark compare?
Does it compare LineairDB vs InnoDB, or can we configure which storage engine to compare?
User should be able to specify the storage engine that they want to compare.

Please use LLM outputs responsibly.

Tatzhiro · 2025-12-24T13:58:25Z

+repl/logs/
+
+# Local test directories
+tony_test/


I will remove it later.

Tatzhiro · 2025-12-24T14:01:16Z

+## Goals
+
+### Functional Requirements
+
+1. **Automatic Group Replication Configuration**
+   - MySQL Group Replication is configured **simply by executing the script**
+   - Leader/Follower role assignment is performed automatically
+
+2. **Node Specification via Configuration File**
+   - Leader address configurable
+   - Follower addresses configurable (multiple allowed)
+   - **Default**: Local MySQL server is configured as the Leader
+
+3. **Script Format**
+   - Implemented in both Shell Script and Python
+   - Easy to read with clear processing flow
+
+### Benchmark Requirements
+
+- Benchmark runs after GR configuration completes
+- Supports `engine=lineairdb` specification
+- Both **benchbase** and **binbench** must complete successfully
+


users dont need to know this, remove

I will remove it later.

Tatzhiro · 2025-12-24T14:03:55Z

+## Completion Checklist
+
+- [x] MySQL Group Replication automatically configured
+- [x] Leader/Follower roles assigned automatically
+- [x] Node specification via configuration file
+- [x] Local MySQL configured as Leader by default
+- [x] Shell script implementation
+- [x] Python API implementation
+- [x] Benchbase with engine=lineairdb succeeds
+- [x] Binbench with engine=lineairdb succeeds


who needs to see this?

Nobody needs, not even me, I will delete this checklist later.

Tatzhiro · 2025-12-24T14:18:05Z

what is the point of this python script? seems to me that it is almost totally unnecessary.

I will try to re-implement without using this script.

Tatzhiro · 2025-12-24T14:19:34Z

+[submodule "third_party/mysql-cluster"]
+	path = third_party/mysql-cluster
+	url = https://github.com/mitou-Kamo/mysql-cluster


do we still need this?

In the current implementation we still need this since I changed it to an interface written by Python to handle cluster construction. But I think this is not necessary anymore since this can be integrated directly to repl/ directory. I would like to discuss with you later.

Tatzhiro · 2025-12-24T14:24:00Z

+./scripts/run_benchmark.sh benchbase ycsb
+./scripts/run_benchmark.sh binbench ycsb


I read run_benchmark.sh, but it seems to me that the first argument should not be benchbase or binbench

Yes you are right, since I fixed run_benchmark.sh but did not update README so I will fix it later.

Tatzhiro · 2026-01-10T02:34:42Z

+
+## Notes
+
+- **LineairDB tables are local only** - they do not replicate across nodes


LineairDB cannot replicate data?

Tatzhiro · 2026-01-10T02:50:45Z

What is this? Please remove if you dont need

Tatzhiro · 2026-01-10T02:58:22Z

I observed a weird behavior running this program.
Is this expected? Why is the systemctl mysqld turned on?

(venv) tatsu@sk030:/work/dpl-sfc/tatsu/LineairDB-storage-engine/repl/scripts$ ps aux | grep mysqld systemd+ 1121392 0.5 0.0 3047524 1167372 ? Ssl Jan08 15:17 mysqld --server-id=2 --log-bin=mysql-bin --gtid-mode=ON --enforce-gtid-consistency=ON --binlog-format=ROW --relay-log=mysql-relay-bin systemd+ 1121967 0.5 0.0 3039332 1161128 ? Ssl Jan08 15:05 mysqld --server-id=3 --log-bin=mysql-bin --gtid-mode=ON --enforce-gtid-consistency=ON --binlog-format=ROW --relay-log=mysql-relay-bin tatsu 1821537 0.0 0.0 10584 2284 pts/8 S+ 11:55 0:00 grep --color=auto mysqld (venv) tatsu@sk030:/work/dpl-sfc/tatsu/LineairDB-storage-engine/repl/scripts$ python cleanup.py -a ╔════════════════════════════════════════════════════════════╗ ║ LineairDB Cluster Cleanup ║ ╚════════════════════════════════════════════════════════════╝ ▶ Cleaning up benchmark databases... No benchmark databases found ▶ Dissolving InnoDB Cluster (if exists)... MySQL Error 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:3306' (111) ▶ Resetting read_only mode on primary... ▶ Removing any remaining containers... ▶ Removing network... ▶ Cleaning up /etc/hosts... ▶ Removing volumes... ▶ Removing configuration... ▶ Removing data directories... ▶ Removing LineairDB logs... ▶ Restarting MySQL... ▶ Waiting for MySQL to be ready... ✓ MySQL is ready ▶ Ensuring read_only is disabled... ╔════════════════════════════════════════════════════════════╗ ║ Cleanup Complete! ║ ╚════════════════════════════════════════════════════════════╝ (venv) tatsu@sk030:/work/dpl-sfc/tatsu/LineairDB-storage-engine/repl/scripts$ ps aux | grep mysqld mysql 1822275 35.3 0.0 1134956 406436 ? Ssl 11:56 0:03 /usr/sbin/mysqld tatsu 1822378 0.0 0.0 10452 2240 pts/8 S+ 11:57 0:00 grep --color=auto mysqld (venv) tatsu@sk030:/work/dpl-sfc/tatsu/LineairDB-storage-engine/repl/scripts$ sudo systemctl status mysql ● mysql.service - MySQL Community Server Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/mysql.service.d └─override.conf Active: active (running) since Sat 2026-01-10 11:56:59 JST; 19s ago Process: 1822267 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS) Main PID: 1822275 (mysqld) Status: "Server is operational" Tasks: 41 (limit: 629145) Memory: 373.2M CPU: 4.020s CGroup: /system.slice/mysql.service └─1822275 /usr/sbin/mysqld Jan 10 11:56:56 sk030 systemd[1]: Starting MySQL Community Server... Jan 10 11:56:59 sk030 systemd[1]: Started MySQL Community Server.

Tatzhiro · 2026-01-10T03:17:53Z

+def generate_cluster_config(
+    num_secondaries: int,
+    output_path: Path,
+    remote_hosts: List[str] = None,
+) -> dict:
+    """
+    Generate cluster configuration.
+
+    Args:
+        num_secondaries: Total number of secondary nodes
+        output_path: Path to save configuration
+        remote_hosts: List of remote host specifications
+
+    Returns:
+        Configuration dictionary
+    """
+    remote_hosts = remote_hosts or []
+
+    # Validate
+    if len(remote_hosts) > num_secondaries:
+        raise ValueError(
+            f"Too many remote hosts ({len(remote_hosts)}) for {num_secondaries} secondaries"
+        )
+
+    num_docker = num_secondaries - len(remote_hosts)
+
+    # Network configuration (for Docker containers)
+    docker_network_name = "mysql-cluster-net"
+    docker_network_subnet = "172.20.0.0/16"
+    docker_base_ip = "172.20.0"
+
+    # MySQL configuration
+    mysql_root_password = MYSQL_PASSWORD
+    mysql_database = "testdb"
+    mysql_user = "clusteruser"
+    mysql_user_password = MYSQL_PASSWORD
+
+    # Primary node (local MySQL)
+    primary = {
+        "node_id": 1,
+        "hostname": socket.gethostname(),
+        "role": "primary",
+        "node_type": "local_systemctl",
+        "host": "127.0.0.1",
+        "port": 3306,
+        "mysql_root_password": mysql_root_password,
+        "server_id": 1,
+    }
+
+    secondaries = []
+    node_id = 2  # Primary is 1, secondaries start at 2
+
+    # Add remote hosts first
+    for host_spec in remote_hosts:
+        host, port, ssh_user = parse_remote_host(host_spec)
+
+        secondary = {
+            "node_id": node_id,
+            "hostname": host,
+            "role": "secondary",
+            "node_type": "remote_host",
+            "host": host,
+            "port": port,
+            "mysql_root_password": mysql_root_password,
+            "ssh_user": ssh_user,
+            "server_id": node_id,
+        }
+        secondaries.append(secondary)
+        node_id += 1
+
+    # Add Docker containers for remaining slots
+    docker_index = 1
+    for _ in range(num_docker):
+        container_name = f"mysql-secondary-{docker_index}"
+        docker_ip = f"{docker_base_ip}.{10 + node_id}"
+        host_port = 33061 + docker_index  # 33062, 33063, ...
+
+        secondary = {
+            "node_id": node_id,
+            "hostname": container_name,
+            "role": "secondary",
+            "node_type": "docker_container",
+            "host": "127.0.0.1",
+            "port": host_port,
+            "mysql_root_password": mysql_root_password,
+            "container_name": container_name,
+            "docker_network": docker_network_name,
+            "docker_ip": docker_ip,
+            "server_id": node_id,
+        }
+        secondaries.append(secondary)
+        node_id += 1
+        docker_index += 1
+
+    # Full configuration
+    config = {
+        "cluster_name": "lineairdb_cluster",
+        "mysql_version": "8.0.43",
+        "mysql_root_password": mysql_root_password,
+        "mysql_database": mysql_database,
+        "mysql_user": mysql_user,
+        "mysql_user_password": mysql_user_password,
+        "docker_network_name": docker_network_name,
+        "docker_network_subnet": docker_network_subnet,
+        "docker_base_ip": docker_base_ip,
+        "docker_image": "mysql-lineairdb-ubuntu:8.0.43",
+        "use_custom_image": True,
+        "primary": primary,
+        "secondaries": secondaries,
+        "num_remote_hosts": len(remote_hosts),
+        "num_docker_containers": num_docker,
+    }
+
+    # Save configuration
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    with open(output_path, "w") as f:
+        json.dump(config, f, indent=2)
+
+    return config


I think the original requirement was to let the users edit config file, and the program reads the config file.
You are doing the other way around, which is ok but weird.
If you want to make a change that goes against requirements, you should ask first.

Add MySQL group replication auto-configuration module

5ac49d3

Tatzhiro requested changes Dec 24, 2025

View reviewed changes

TonyLi15 added 2 commits December 26, 2025 15:22

Refactor shell scripts to python and Remove the submodule mysql-cluster

6d37c9c

Fix cluster setup timeout and benchmark result path extraction

2108344

Tatzhiro requested changes Jan 10, 2026

View reviewed changes

		./scripts/run_benchmark.sh benchbase ycsb
		./scripts/run_benchmark.sh binbench ycsb


		## Notes

		- LineairDB tables are local only - they do not replicate across nodes

Conversation

TonyLi15 commented Dec 23, 2025

Uh oh!

Tatzhiro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants