fix(server): INFO # Keyspace lists all dbs across all shards; GRAPH.LIST unions all shards#280
Merged
Merged
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
📝 WalkthroughWalkthroughINFO now aggregates non-empty logical database keyspace counters across shards, while multi-shard GRAPH.LIST scatters requests and returns a sorted, deduplicated union of graph names. ChangesListing parity
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant INFOHandler
participant ShardCoordinator
participant RemoteShard
Client->>INFOHandler: INFO
INFOHandler->>ShardCoordinator: aggregate keyspace counters
ShardCoordinator->>RemoteShard: KeyspaceStats
RemoteShard-->>ShardCoordinator: per-database counters
ShardCoordinator-->>INFOHandler: summed keyspace
INFOHandler-->>Client: INFO with non-empty db entries
sequenceDiagram
participant Client
participant GRAPHLISTHandler
participant RemoteShard
participant GraphMerger
Client->>GRAPHLISTHandler: GRAPH.LIST
GRAPHLISTHandler->>RemoteShard: GraphCommand on each remote shard
RemoteShard-->>GRAPHLISTHandler: graph arrays
GRAPHLISTHandler->>GraphMerger: local and remote responses
GraphMerger-->>Client: sorted deduplicated graph union
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…IST unions all shards Listing parity (task #26): two commands silently reported a single shard's local view as if it were the whole server. INFO # Keyspace previously printed one hardcoded `db0:` line holding the SELECTED db's LOCAL-shard key count — `SELECT 2; SET k v; INFO` reported the db-2 count as db0, other dbs were invisible, and at --shards N the other shards' keys were uncounted. Now: - New `ShardMessage::KeyspaceStats` typed-reply scatter (same pattern as GetKeysInSlot/AofFold): each shard returns per-db (keys, expires) from O(#dbs) counter reads — no key iteration. - `coordinate_keyspace_info` (shard/coordinator.rs) sums element-wise across shards, degrades to partial results on a dead channel. - `info_with_keyspace` (command/connection.rs) rewrites the # Keyspace section listing every NON-EMPTY logical db as `db{i}:keys=K,expires=E,avg_ttl=0` (Redis semantics). - Wired on all three dispatch paths: monoio dispatch, tokio handler_sharded dispatch, and handler_single (single shard = truth, no scatter needed). GRAPH.LIST previously listed only the connection shard's graphs (~1/N, since a graph lives on the shard that owns its name). Now scatters a GraphCommand to every shard and returns the sorted, deduplicated union via `merge_graph_list_responses` (skips error/non-array frames so one sick shard degrades instead of failing the command). Wired on both monoio and tokio sharded write paths. Verified: lib tests 4109 pass, fmt + clippy clean (default and runtime-tokio,jemalloc), smoke-tested at --shards 1 and --shards 4 (8 graphs created round-robin all listed; SELECT 2/3/5 writes all reported with cross-shard sums). author: Tin Dang
TinDang97
force-pushed
the
feat/listing-parity
branch
from
63b79fd to
758e95a
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/server/conn/handler_single.rs (1)
875-904: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winRun the keyspace-aware INFO path even without replication state.
When
repl_stateisNone, the code falls through at Line 903 to generic dispatch, so single-shard INFO still reports only the selected database asdb0and hides other non-empty databases. Build the keyspace response unconditionally; append replication details only whenrepl_stateis present.Proposed fix
if cmd.eq_ignore_ascii_case(b"INFO") { - if let Some(ref rs) = repl_state { - // build keyspace response - // append replication response - continue; - } - // Fall through to normal dispatch if no repl_state + // Build the keyspace-aware response unconditionally. + // Append replication details only when available. + continue; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/server/conn/handler_single.rs` around lines 875 - 904, Make the keyspace-aware INFO handling unconditional instead of gating it on repl_state: in the INFO branch, always build the per-database keyspace, call conn_cmd::info_with_keyspace, and append replication details only when repl_state is Some (using try_read as currently). Remove the fallback to generic dispatch so INFO reports all non-empty databases even without replication state.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/server/conn/handler_monoio/dispatch.rs`:
- Around line 641-652: Gate the cross-shard call to coordinate_keyspace_info in
the INFO handling path so it runs only when the requested sections include
Keyspace. Reuse the section parsing logic from conn_cmd::info, covering explicit
Keyspace, default, and all-section requests; otherwise skip coordination and
pass an empty or absent keyspace result to info_with_keyspace while preserving
existing responses.
In `@src/server/conn/handler_monoio/write.rs`:
- Around line 857-891: Update the GRAPH.LIST fan-out handling in the sharded
branch to await all remote replies concurrently instead of serially iterating
over receivers. Use FuturesUnordered, join_all, or an equivalent concurrent
pattern around recv_reply_bounded so each shard’s timeout runs in parallel,
while preserving successful partial replies in remotes before merging with the
local response.
In `@src/server/conn/handler_sharded/write.rs`:
- Around line 764-798: GRAPH.LIST waits for remote shard replies sequentially,
allowing one stalled shard to delay all others. Update the reply-collection loop
in the sharded GRAPH.LIST handling block to await all receivers concurrently,
such as by joining or spawning bounded receive futures, then collect each
successful response into remotes before merging with the local result via
merge_graph_list_responses.
In `@src/shard/coordinator.rs`:
- Around line 1814-1822: Update the receiver aggregation loop in the
coordinator’s keyspace stats method to await all shard replies concurrently
rather than calling recv_reply_bounded(rx).await serially. Spawn or join one
bounded receive future per receiver, then aggregate each successful result into
totals so the operation completes within a single XSHARD_REPLY_TIMEOUT.
---
Outside diff comments:
In `@src/server/conn/handler_single.rs`:
- Around line 875-904: Make the keyspace-aware INFO handling unconditional
instead of gating it on repl_state: in the INFO branch, always build the
per-database keyspace, call conn_cmd::info_with_keyspace, and append replication
details only when repl_state is Some (using try_read as currently). Remove the
fallback to generic dispatch so INFO reports all non-empty databases even
without replication state.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 281da787-ff91-4582-b018-b2850f84b743
📒 Files selected for processing (14)
CHANGELOG.mdsrc/command/connection.rssrc/command/graph/graph_read.rssrc/command/graph/mod.rssrc/server/conn/handler_monoio/dispatch.rssrc/server/conn/handler_monoio/mod.rssrc/server/conn/handler_monoio/write.rssrc/server/conn/handler_sharded/dispatch.rssrc/server/conn/handler_sharded/mod.rssrc/server/conn/handler_sharded/write.rssrc/server/conn/handler_single.rssrc/shard/coordinator.rssrc/shard/dispatch.rssrc/shard/spsc_handler.rs
Comment on lines
+641
to
+652
| // # Keyspace parity: per-db (keys, expires) summed across ALL shards — | ||
| // previously the section reported the selected db's LOCAL count as db0 | ||
| // (other dbs invisible, other shards uncounted). | ||
| let keyspace = crate::shard::coordinator::coordinate_keyspace_info( | ||
| ctx.shard_id, | ||
| ctx.num_shards, | ||
| &ctx.dispatch_tx, | ||
| &ctx.spsc_notifiers, | ||
| ) | ||
| .await; | ||
| let response_text = crate::shard::slice::with_shard_db(conn.selected_db, |db| { | ||
| let resp_frame = conn_cmd::info_readonly(db, cmd_args); | ||
| let resp_frame = conn_cmd::info_with_keyspace(db, cmd_args, &keyspace); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Only scatter when the requested INFO section includes Keyspace.
Line 644 runs the cross-shard gather for INFO server, INFO clients, etc.; info_with_keyspace then returns the unchanged local response when # Keyspace is absent. Gate this call using the same section parsing as conn_cmd::info, including the default/all forms, so unrelated diagnostics do not incur shard timeouts.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/server/conn/handler_monoio/dispatch.rs` around lines 641 - 652, Gate the
cross-shard call to coordinate_keyspace_info in the INFO handling path so it
runs only when the requested sections include Keyspace. Reuse the section
parsing logic from conn_cmd::info, covering explicit Keyspace, default, and
all-section requests; otherwise skip coordination and pass an empty or absent
keyspace result to info_with_keyspace while preserving existing responses.
Comment on lines
+857
to
+891
| if ctx.num_shards > 1 && cmd.eq_ignore_ascii_case(b"GRAPH.LIST") { | ||
| let mut receivers = Vec::with_capacity(ctx.num_shards - 1); | ||
| for target in 0..ctx.num_shards { | ||
| if target == ctx.shard_id { | ||
| continue; | ||
| } | ||
| let (reply_tx, reply_rx) = crate::runtime::channel::oneshot(); | ||
| let msg = crate::shard::dispatch::ShardMessage::GraphCommand { | ||
| command: std::sync::Arc::new(frame.clone()), | ||
| reply_tx, | ||
| }; | ||
| let _ = crate::shard::coordinator::spsc_send( | ||
| &ctx.dispatch_tx, | ||
| ctx.shard_id, | ||
| target, | ||
| msg, | ||
| &ctx.spsc_notifiers, | ||
| ) | ||
| .await; | ||
| receivers.push(reply_rx); | ||
| } | ||
| let mut remotes = Vec::with_capacity(receivers.len()); | ||
| for rx in receivers { | ||
| if let Ok(f) = crate::shard::coordinator::recv_reply_bounded(rx).await { | ||
| remotes.push(f); | ||
| } | ||
| } | ||
| let local = crate::shard::slice::with_shard(|s| { | ||
| crate::command::graph::dispatch_graph_read(&s.graph_store, cmd, cmd_args, None) | ||
| }); | ||
| responses.push(crate::command::graph::merge_graph_list_responses( | ||
| local, &remotes, | ||
| )); | ||
| return true; | ||
| } |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
grep -n '^futures' Cargo.toml
rg -n 'join_all|FuturesUnordered|select_all' src --type rust
ast-grep run --pattern 'fn race2($$$) { $$$ }' --lang rust src/runtimeRepository: pilotspace/moon
Length of output: 1359
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant files first
ast-grep outline src/server/conn/handler_monoio/write.rs --view expanded
ast-grep outline src/shard/coordinator.rs --view expanded
ast-grep outline src/runtime/channel.rs --view expanded
ast-grep outline src/server/conn/blocking.rs --view expanded
# Inspect the graph-list branch and the bounded receive helper
sed -n '840,910p' src/server/conn/handler_monoio/write.rs
rg -n 'recv_reply_bounded|XSHARD_REPLY_TIMEOUT|FuturesUnordered|join_all|select_all' src/shard src/runtime src/server/conn --type rust
sed -n '1,220p' src/shard/coordinator.rs
sed -n '230,390p' src/runtime/channel.rs
sed -n '200,290p' src/server/conn/blocking.rsRepository: pilotspace/moon
Length of output: 30386
Await the GRAPH.LIST replies concurrently.
The current serial for rx in receivers loop can turn K stalled shards into K × XSHARD_REPLY_TIMEOUT total latency. Use FuturesUnordered or join_all so the per-shard timeouts run in parallel and partial results return promptly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/server/conn/handler_monoio/write.rs` around lines 857 - 891, Update the
GRAPH.LIST fan-out handling in the sharded branch to await all remote replies
concurrently instead of serially iterating over receivers. Use FuturesUnordered,
join_all, or an equivalent concurrent pattern around recv_reply_bounded so each
shard’s timeout runs in parallel, while preserving successful partial replies in
remotes before merging with the local response.
Comment on lines
+764
to
+798
| if ctx.num_shards > 1 && cmd.eq_ignore_ascii_case(b"GRAPH.LIST") { | ||
| let mut receivers = Vec::with_capacity(ctx.num_shards - 1); | ||
| for target in 0..ctx.num_shards { | ||
| if target == ctx.shard_id { | ||
| continue; | ||
| } | ||
| let (reply_tx, reply_rx) = crate::runtime::channel::oneshot(); | ||
| let msg = crate::shard::dispatch::ShardMessage::GraphCommand { | ||
| command: std::sync::Arc::new(frame.clone()), | ||
| reply_tx, | ||
| }; | ||
| let _ = crate::shard::coordinator::spsc_send( | ||
| &ctx.dispatch_tx, | ||
| ctx.shard_id, | ||
| target, | ||
| msg, | ||
| &ctx.spsc_notifiers, | ||
| ) | ||
| .await; | ||
| receivers.push(reply_rx); | ||
| } | ||
| let mut remotes = Vec::with_capacity(receivers.len()); | ||
| for rx in receivers { | ||
| if let Ok(f) = crate::shard::coordinator::recv_reply_bounded(rx).await { | ||
| remotes.push(f); | ||
| } | ||
| } | ||
| let local = crate::shard::slice::with_shard(|s| { | ||
| crate::command::graph::dispatch_graph_read(&s.graph_store, cmd, cmd_args, None) | ||
| }); | ||
| responses.push(crate::command::graph::merge_graph_list_responses( | ||
| local, &remotes, | ||
| )); | ||
| return true; | ||
| } |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant files and symbols first.
ast-grep outline src/server/conn/handler_sharded/write.rs --view expanded | sed -n '1,220p'
printf '\n---\n'
ast-grep outline src/server/conn/handler_monoio/write.rs --view expanded | sed -n '1,240p'
printf '\n---\n'
rg -n "recv_reply_bounded|XSHARD_REPLY_TIMEOUT|GraphCommand|GRAPH.LIST|dispatch_graph_read|merge_graph_list_responses" src/server src/shard src/command -SRepository: pilotspace/moon
Length of output: 9011
🏁 Script executed:
ast-grep outline src/server/conn/handler_sharded/write.rs --view expanded | sed -n '1,220p'
printf '\n---\n'
ast-grep outline src/server/conn/handler_monoio/write.rs --view expanded | sed -n '1,240p'
printf '\n---\n'
rg -n "recv_reply_bounded|XSHARD_REPLY_TIMEOUT|GraphCommand|GRAPH.LIST|dispatch_graph_read|merge_graph_list_responses" src/server src/shard src/command -SRepository: pilotspace/moon
Length of output: 9011
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '746,802p' src/server/conn/handler_sharded/write.rs
printf '\n---\n'
sed -n '839,893p' src/server/conn/handler_monoio/write.rs
printf '\n---\n'
sed -n '191,230p' src/shard/coordinator.rsRepository: pilotspace/moon
Length of output: 6677
Collect shard replies concurrently. The for loop waits on recv_reply_bounded(rx).await one shard at a time, so a stalled early shard can delay already-ready replies and stretch GRAPH.LIST latency by roughly N × XSHARD_REPLY_TIMEOUT.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/server/conn/handler_sharded/write.rs` around lines 764 - 798, GRAPH.LIST
waits for remote shard replies sequentially, allowing one stalled shard to delay
all others. Update the reply-collection loop in the sharded GRAPH.LIST handling
block to await all receivers concurrently, such as by joining or spawning
bounded receive futures, then collect each successful response into remotes
before merging with the local result via merge_graph_list_responses.
Comment on lines
+1814
to
+1822
| for rx in receivers { | ||
| if let Ok(stats) = recv_reply_bounded(rx).await { | ||
| for (i, (k, e)) in stats.into_iter().enumerate() { | ||
| if let Some(t) = totals.get_mut(i) { | ||
| t.0 += k; | ||
| t.1 += e; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Await shard replies concurrently.
Line 1814 waits receivers serially, so each unavailable shard adds another XSHARD_REPLY_TIMEOUT; INFO Keyspace can block for (shards - 1) × timeout despite already-available replies. Drive all bounded receives concurrently so partial results return within one timeout.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/shard/coordinator.rs` around lines 1814 - 1822, Update the receiver
aggregation loop in the coordinator’s keyspace stats method to await all shard
replies concurrently rather than calling recv_reply_bounded(rx).await serially.
Spawn or join one bounded receive future per receiver, then aggregate each
successful result into totals so the operation completes within a single
XSHARD_REPLY_TIMEOUT.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Listing parity (task #26): two commands silently reported a single shard's local view as if it were the whole server.
INFO# Keyspace— all dbs × all shardsPreviously printed one hardcoded
db0:line holding the SELECTED db's LOCAL-shard key count —SELECT 2; SET k v; INFOreported the db-2 count as db0, other dbs were invisible, and at--shards Nthe other shards' keys were uncounted. Now:ShardMessage::KeyspaceStatstyped-reply scatter (same pattern as GetKeysInSlot/AofFold): each shard returns per-db(keys, expires)from O(#dbs) counter reads — no key iteration.coordinate_keyspace_infosums element-wise across shards; degrades to partial results on a dead channel.info_with_keyspacelists every NON-EMPTY logical db asdb{i}:keys=K,expires=E,avg_ttl=0(Redis semantics).GRAPH.LIST— all shardsPreviously listed only the connection shard's graphs (~1/N — a graph lives on the shard owning its name). Now scatters a GraphCommand to every shard and returns the sorted, deduplicated union via
merge_graph_list_responses(skips error/non-array frames so one sick shard degrades instead of failing the command). Both monoio and tokio sharded paths.Verification
runtime-tokio,jemalloc)--shards 1and--shards 4: 8 graphs created round-robin all listed;SELECT 2/3/5writes all reported with cross-shard sumsinfo_with_keyspace_lists_every_nonempty_db,merge_graph_list_unions_sorts_dedups_and_skips_errorsSummary by CodeRabbit
INFOkeyspace reporting to include every non-empty logical database and aggregate keys and expirations across shards.GRAPH.LIST --shards Nto return a sorted, deduplicated list combining graphs from all shards.