Skip to content

[Store] Throttle repeated no-master client logs via LOG_EVERY_T (follow-up to #2484) - #1

Open
silas-scitix wants to merge 1 commit into
mainfrom
feat/throttle-no-master-logs-glog
Open

[Store] Throttle repeated no-master client logs via LOG_EVERY_T (follow-up to #2484)#1
silas-scitix wants to merge 1 commit into
mainfrom
feat/throttle-no-master-logs-glog

Conversation

@silas-scitix

@silas-scitix silas-scitix commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

What

This is a follow-up to kvcache-ai#2484. During a persistent no-master window, the
storage heartbeat loop (StorageHeartbeatThreadMain) and the leader
monitor loop (LeaderMonitorThreadMain) re-emit the same failure log on
every retry (fail_ping_interval_ms / kErrorRetryInterval, ~1s). With a
large number of clients this becomes a log storm that drowns out other
diagnostics and grows the log volume linearly with the client count.

This wraps each repeating no-master log site with glog's built-in
LOG_EVERY_T(SEV, 30), so each call site emits at most once per 30
wall-seconds per process while keeping the original message text,
severity, and the first occurrence. No throttle helper is hand-rolled;
the built-in glog macro is used directly.

Sites changed (mooncake-store/src/client_service.cpp)

StorageHeartbeatThreadMain (heartbeat ping/reconnect loop):

  • "Failed to ping master"
  • "Failed to get new master view"
  • "No active master view is published yet"
  • "Failed to connect to master"
  • "Failed to ping master for N times (non-HA)"
  • "Reconnect failed to"

LeaderMonitorThreadMain:

  • "Failed to wait for leader view change"
  • "Failed to switch to leader"

Not changed

The one-shot connect logs in ConnectToMaster are intentionally left
unthrottled (they fire once per connect attempt, not in a retry loop).

Behavior

Behavior-neutral except for log frequency. Retry intervals, sleeps,
ping_fail_count, and reconnect control flow are untouched; only the
LOG() call at each repeating site is swapped for the throttled glog
macro.

Testing

  • Built GREEN on a fresh upstream main checkout (7e9bdde).
  • scripts/code_format.sh --check passes (clang-format clean on the
    changed file).
  • ctest: no regression on the directly relevant tests
    (non_ha_reconnect_test, ha_recovery_test,
    ha_backend_availability_test, client_integration_test,
    master_service_test, client_metrics_test).

…ow-up to kvcache-ai#2484)

During a persistent no-master window, the storage heartbeat loop
(StorageHeartbeatThreadMain) and the leader monitor loop
(LeaderMonitorThreadMain) re-emit the same failure log on every retry
(fail_ping_interval_ms / kErrorRetryInterval, ~1s). With many clients
this becomes a log storm that drowns out other diagnostics and grows
the log volume linearly with the client count. This is a follow-up to
kvcache-ai#2484, which introduced the heartbeat/leader-monitor retry loops.

Wrap each repeating no-master log site with glog's built-in
LOG_EVERY_T(SEV, 30) so a given call site emits at most once per 30
wall-seconds per process while preserving the original message text,
severity, and the first occurrence. No throttle helper is hand-rolled;
the built-in glog macro is used directly.

Sites changed (mooncake-store/src/client_service.cpp):
  StorageHeartbeatThreadMain: "Failed to ping master",
  "Failed to get new master view", "No active master view is published
  yet", "Failed to connect to master", "Failed to ping master for N
  times (non-HA)", "Reconnect failed to".
  LeaderMonitorThreadMain: "Failed to wait for leader view change",
  "Failed to switch to leader".

The one-shot connect logs in ConnectToMaster are intentionally left
unthrottled (they fire once per connect attempt, not in a retry loop).

Behavior is unchanged except for log frequency: retry intervals,
sleeps, ping_fail_count, and reconnect control flow are untouched.
Build is GREEN and ctest shows no regression on the heartbeat,
reconnect, and HA recovery tests; clang-format check passes.
@silas-scitix
silas-scitix force-pushed the feat/throttle-no-master-logs-glog branch from 442e804 to 93375d8 Compare June 23, 2026 09:59
@silas-scitix silas-scitix changed the title [Store] Throttle repeating no-master heartbeat logs with LOG_EVERY_T [Store] Throttle repeated no-master client logs via LOG_EVERY_T (follow-up to #2484) Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant