Skip to content

Fix flaky ClusterMigrateRangeIndexCheckpointThenRecover: flush AOF before graceful restart#1959

Open
tiagonapoli wants to merge 3 commits into
mainfrom
tiagonapoli/fix-range-index-tests
Open

Fix flaky ClusterMigrateRangeIndexCheckpointThenRecover: flush AOF before graceful restart#1959
tiagonapoli wants to merge 3 commits into
mainfrom
tiagonapoli/fix-range-index-tests

Conversation

@tiagonapoli

Copy link
Copy Markdown
Collaborator

Problem

ClusterMigrateRangeIndexCheckpointThenRecover (in Garnet.test.cluster.migrate.rangeindex) intermittently fails with "post-checkpoint RI.SET should survive recovery".

Root cause

The test does a post-checkpoint RI.SET and then immediately restarts the target node. That write is acknowledged to the client but, at CommitFrequencyMs=0 (async auto-commit), is not yet durably flushed to the AOF. A graceful shutdown (GarnetServer.DisposeStoreWrapper.Dispose) cancels the background committer and does not flush the AOF, so the acked-but-uncommitted tail is lost on recovery — the test then can't find the post_ckpt value.

This mirrors the durability contract every other AOF-recovery test relies on (they all issue COMMITAOF / commitWait / CommitAOFAsync before restart); this test omitted it.

Fix (test-only)

Add an opt-in ensureAofFlush parameter to ClusterTestContext.ShutdownNode / RestartNode that issues a synchronous Store.CommitAOFAsync before disposing the node. The RangeIndex migrate test's RestartNode helper passes ensureAofFlush: true so all its restarts flush pending AOF commits first.

Validation

  • Builds clean (0 warnings).
  • Previously-flaky test passed 7/7 locally.
  • Docker CPU-throttled repro fleet run to confirm the intermittent failure no longer reproduces (in progress).

The cluster test ClusterMigrateRangeIndexCheckpointThenRecover intermittently
failed because a post-checkpoint RI.SET was acked but not durably committed to
the AOF before the node was gracefully restarted. Graceful Dispose does not
flush the AOF, so the write was lost on recovery.

Add an opt-in ensureAofFlush parameter to ClusterTestContext.ShutdownNode /
RestartNode that issues a synchronous CommitAOFAsync before disposing the node,
and use it from the RangeIndex migrate test's RestartNode helper so all its
restarts flush pending AOF commits first.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a8557ca8-3a4c-487c-bbf1-dd45e03caca5
@tiagonapoli
tiagonapoli marked this pull request as ready for review July 21, 2026 18:04
Copilot AI review requested due to automatic review settings July 21, 2026 18:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses intermittent failure in the cluster RangeIndex migration recovery test by ensuring pending AOF work is durably committed before a “graceful” node restart during the test.

Changes:

  • Added an opt-in ensureAofFlush parameter to ClusterTestContext.ShutdownNode(...) and RestartNode(...) to force an AOF commit prior to disposal.
  • Updated the RangeIndex migrate test helper to restart nodes with ensureAofFlush: true.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
test/cluster/Garnet.test.cluster/ClusterTestContext.cs Adds optional AOF commit before shutdown/restart to make test restarts durable.
test/cluster/Garnet.test.cluster.migrate.rangeindex/ClusterRangeIndexMigrateTests.cs Uses the new option to flush AOF before restarting the target node in the flaky test.

Comment thread test/cluster/Garnet.test.cluster/ClusterTestContext.cs
@vazois
vazois self-requested a review July 21, 2026 18:22
@vazois vazois self-assigned this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants