fix(replication): stream SELECT for multi-db writes + serve SELECT on replicas#281
Conversation
… replicas HIGH-2 (tasks #22 + #23): the replica-side drain already understood in-stream SELECT context (drain_replicated_commands binds each command to the selected db), but the master never emitted it — every replicated write applied to the replica's db 0 regardless of which db it executed in on the master. And a client could not even READ a replica's non-zero dbs: SELECT is flagged W in the metadata table, so the READONLY guard rejected it. Master side: - ReplicationState grows per-shard `stream_db: Vec<AtomicI64>` (-1 = unknown): the logical-db context of the LAST data command recorded into the shard's replication backlog. - New `record_local_write_db(ctx, db, bytes)` prepends a `SELECT <db>` record via the existing `record_local_write` primitive whenever the writing connection's db differs from the stream context. Both records stay in the same synchronous stretch as the mutation (exactly-once contract unchanged; SELECT bytes advance the offset like any stream bytes). - All db-scoped record sites switched to the db-aware variant: the three KV legs (main, MOVE, COPY), the MULTI/EXEC body loop, the GRAPH.* WAL-record leg, and the FT.* index-def leg. The TEMPORAL.INVALIDATE-AT leg keeps the primitive (db-agnostic record, leaves the context untouched — which stays truthful). - FULLRESYNC snapshot capture resets the shard's stream_db to -1 in the SAME synchronous stretch as the RDB capture (Redis's `slaveseldb = -1` idiom): every byte at offset >= snapshot_offset starts from "db unknown", so the first post-snapshot write re-establishes the context with an explicit SELECT and a fresh replica (whose drain starts at db 0) can never bind a write to the wrong db. Redundant re-SELECTs for already-attached replicas are idempotent. Replica side: - `ReplicaTaskConfig::stream_db` (AtomicUsize) preserves the drain's db context ACROSS reconnects: a +CONTINUE partial resync replays backlog bytes that only carry SELECT at db CHANGES — if the stream was in db N when the link dropped, the resumed bytes have no fresh SELECT and a 0-reset drain would misapply them to db 0. Seeded into stream_commands on entry, persisted after every drain, reset to 0 on +FULLRESYNC. In-memory only: a replica process restart starts at offset 0 and always full-resyncs. - READONLY guards (monoio dispatch, tokio sharded dispatch, handler_single) now serve SELECT on replicas — connection-state only, Redis parity. New e2e replica_applies_multi_db_stream (RED before the fix — failed at the READONLY SELECT assertion, then would misapply db-2 writes to db 0): db-0 write flows, SELECT 2 + SET on the master, db-0 hop-back sentinel; asserts the db-2 write lands in db 2 on the replica, did NOT leak into db 0, and the db-0 sentinel did not leak into db 2. Verification: lib 4111 green, replication_streaming 5/5 (incl. new test), replication_graph 3/3, replication_hardening 5/5, tokio replication_test 5/5, fmt clean, clippy clean on default and runtime-tokio,jemalloc. author: Tin Dang
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Warning Review limit reached
Next review available in: 2 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (12)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2 participants
Summary
Multi-db replication correctness (HIGH-2, tasks #22 + #23): the replica-side drain already understood in-stream
SELECTcontext, but the master never emitted it — every replicated write applied to the replica's db 0 regardless of which db it executed in on the master. And a client couldn't even read a replica's non-zero dbs:SELECTis flagged W in the metadata table, so the READONLY guard rejected it.Master streams
SELECTReplicationStategrows per-shardstream_db(-1= unknown): the db context of the last data command recorded into the shard's backlog.record_local_write_db(ctx, db, bytes)prepends aSELECT <db>record whenever the writing connection's db differs — both records in the same synchronous stretch as the mutation (exactly-once contract unchanged).stream_dbto-1in the same synchronous stretch as the RDB capture (Redis'sslaveseldb = -1idiom): post-snapshot bytes always re-establish context, so a fresh replica (drain starts at db 0) can never bind a write to the wrong db.Replica preserves context across
+CONTINUEResumed backlog bytes only carry
SELECTat db changes — if the stream was in db N when the link dropped, a 0-reset drain would misapply resumed bytes to db 0.ReplicaTaskConfig::stream_dbnow persists the drain-side db across reconnects (reset to 0 on FULLRESYNC; in-memory only — a process restart starts at offset 0 and always full-resyncs).Replicas serve
SELECT(#23)All three dispatch paths (monoio, tokio sharded, single) exempt SELECT from the READONLY guard — connection-state only, Redis parity.
Verification
replica_applies_multi_db_stream— RED before the fix (failed first at the READONLY SELECT assertion; the master also misapplied db-2 writes to db 0), GREEN after. Asserts the db-2 write lands in db 2, did not leak into db 0, and the db-0 sentinel did not leak into db 2.