Skip to content

fix: Fix Codis lossless master switch offset alignment - #3268

Open
Mixficsol wants to merge 5 commits into
3.5from
hotfix/check_master
Open

fix: Fix Codis lossless master switch offset alignment#3268
Mixficsol wants to merge 5 commits into
3.5from
hotfix/check_master

Conversation

@Mixficsol

Copy link
Copy Markdown
Collaborator

解决了什么问题

高写入 QPS 下主动切主会失败并丢数据。根因是「切主前确认从库追平」的判断口径错误:代码拿新主自己写 binlog 的 producer 位点去比老主的 producer 位点,而两者是各自独立的坐标系(各自按 binlog-file-size 分文件、各自 rollover),newFileNum > oldFileNum 就误判「已追上」。从库文件号大往往只是 binlog-file-size 调小造成的假象,结果没真追平就切主——老主 pause 期间的尾部写入丢失,老主还被迫全量重做。

怎么解决的

把「确认从库追平」的信号从「比两节点各自的 producer 位点」改成「老主停写后,看老主视角下该从库的 lag 收敛到阈值内」,这才是同一坐标系下可信的追平信号。

1. sentinel.go — 修正 lag 解析

  • 新增 parseSlaveLag:正确解析 pika 的 lag=(db0:N) 格式,多 db 取最大值(任一 db 没追平就算没追平),兼容裸整数,full syncing/not syncing 等非数字状态返回哨兵值(永不判为追平)。
  • 修复隐藏 bug:InfoSlave.UnmarshalJSON 原用 strconv.Atoi("(db0:0)") 恒失败,导致 Lag 字段一直为 0;改用 parseSlaveLag

2. topom_group.go — 重写追平判断

  • 删除错误的 binlogCaughtUp(比较两节点各自独立的 producer 位点,丢数据根源)。
  • 新增 getSlaveLagFromMaster:读老主 info replication,按地址定位新主,返回老主视角下该从库的 lag;新主不在老主 slave 列表时判为「未追平」,绝不当作 lag 0。
  • 重写 waitCatchUpBeforePromote:老主 pause-write 停写 → 轮询老主视角下新主的 lag → lag <= threshold 才算追平 → 超时按默认策略 abort 并恢复老主写入。

3. client_test.go — 补充单测

  • 新增 TestParseSlaveLagTestMasterInfoReplicationLag,原 master 用例补 Lag 断言。

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9b1cd3d0-54d5-4a9d-ab83-d9cd02d38361

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch hotfix/check_master

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the ☢️ Bug Something isn't working label Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

☢️ Bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant