Skip to content

etcd(ticdc): update etcd cluster member correctly (#12646)#12650

Merged
ti-chi-bot[bot] merged 6 commits into
pingcap:release-8.5from
ti-chi-bot:cherry-pick-12646-to-release-8.5
May 29, 2026
Merged

etcd(ticdc): update etcd cluster member correctly (#12646)#12650
ti-chi-bot[bot] merged 6 commits into
pingcap:release-8.5from
ti-chi-bot:cherry-pick-12646-to-release-8.5

Conversation

@ti-chi-bot

Copy link
Copy Markdown
Member

This is an automated cherry-pick of #12646

What problem does this PR solve?

Issue Number: close #12368

What is changed and how it works?

When updating etcd client URLs, remove the client that is not a member of the etcd cluster.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
tiup playground $VERSION --db 1 --kv 1 --pd 3 --ticdc 1 --tiflash 0 --without-monitor
tiup cdc:$VERSION cli changefeed create --sink-uri 'blackhole://'
tiup playground scale-out --pd 1
tiup playground display
tiup playground scale-in --pid 153389 # pick any one from the display

Print only once

[2026/05/21 10:03:39.588 +00:00] [INFO] [client.go:447] ["update endpoints"] [numChange=3->4] [lastEndpoints="[http://127.0.0.1:2382,http://127.0.0.1:2379,http://127.0.0.1:2384]"] [endpoints="[http://127.0.0.1:44765,http://127.0.0.1:2382,http://127.0.0.1:2384,http://127.0.0.1:2379]"]

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fixed the bug of using an incorrect etcd client when scaling PD nodes

wk989898 and others added 6 commits May 25, 2026 05:01
Signed-off-by: wk989898 <nhsmwk@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: wk989898 <nhsmwk@gmail.com>
Signed-off-by: wk989898 <nhsmwk@gmail.com>
Signed-off-by: wk989898 <nhsmwk@gmail.com>
Signed-off-by: wk989898 <nhsmwk@gmail.com>
@ti-chi-bot ti-chi-bot added lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. type/cherry-pick-for-release-8.5 This PR is cherry-picked to release-8.5 from a source PR. labels May 25, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the etcd client configuration by removing the AutoSyncInterval, making the isHealthy function private, and refining the healthyChecker.update logic to properly close and remove stale client connections. Feedback was provided regarding the update function, specifically pointing out that an empty endpoint list could lead to unnecessary connection churn and suggesting a guard clause to prevent clearing all clients during transient failures.

Comment thread pkg/etcd/client.go
}

func (checker *healthyChecker) update(eps []string) {
updateEps := make(map[string]struct{}, len(eps))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If syncUrls fails to retrieve the etcd member list (e.g., due to a transient network error or leader election), it returns an empty slice. In the current implementation, this causes the update function to clear all existing clients from the healthyChecker and close their connections. When syncUrls succeeds in a subsequent iteration, all clients will be recreated, leading to unnecessary connection churn and a temporary loss of health monitoring for existing members. Adding a guard to skip the update when eps is empty ensures continuity during transient etcd control plane issues.

	if len(eps) == 0 {
		return
	}
	updateEps := make(map[string]struct{}, len(eps))

@codecov

codecov Bot commented May 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 18 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release-8.5@1d4e139). Learn more about missing BASE report.

Additional details and impacted files
Components Coverage Δ
cdc 57.6522% <0.0000%> (?)
dm 49.0171% <0.0000%> (?)
engine 50.7219% <0.0000%> (?)
Flag Coverage Δ
cdc 57.6522% <0.0000%> (?)
unit 53.4621% <0.0000%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             release-8.5     #12650   +/-   ##
================================================
  Coverage               ?   53.4621%           
================================================
  Files                  ?       1026           
  Lines                  ?     137935           
  Branches               ?          0           
================================================
  Hits                   ?      73743           
  Misses                 ?      58681           
  Partials               ?       5511           
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ti-chi-bot

ti-chi-bot Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added approved cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels May 25, 2026
@wk989898

Copy link
Copy Markdown
Collaborator

/retest

1 similar comment
@wk989898

Copy link
Copy Markdown
Collaborator

/retest

@ti-chi-bot ti-chi-bot Bot merged commit 55dc97f into pingcap:release-8.5 May 29, 2026
27 checks passed
@ti-chi-bot ti-chi-bot Bot deleted the cherry-pick-12646-to-release-8.5 branch May 29, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved cherry-pick-approved Cherry pick PR approved by release team. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. type/cherry-pick-for-release-8.5 This PR is cherry-picked to release-8.5 from a source PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants