[ISSUE #10518] Add client config to ignore corrupted local offset files#10568
[ISSUE #10518] Add client config to ignore corrupted local offset files#10568wang-jiahua wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in client configuration to allow consumers to ignore corrupted local offset backup files (offsets.json.bak) instead of failing startup, addressing the scenario where offset persistence can be non-critical and corruption can occur (e.g., truncation).
Changes:
- Added
ClientConfig.localOffsetStoreIgnoreCorrupted(defaultfalse) with accessor methods. - Updated
LocalFileOffsetStore.readLocalOffsetBak()to optionally skip corrupted.bakcontent and continue with an empty offset table.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| client/src/main/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStore.java | Adds opt-in behavior to ignore corrupted local offset backup files during load. |
| client/src/main/java/org/apache/rocketmq/client/ClientConfig.java | Introduces a new client flag controlling corrupted local offset file handling. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #10568 +/- ##
=============================================
- Coverage 48.28% 48.15% -0.14%
+ Complexity 13445 13402 -43
=============================================
Files 1378 1378
Lines 100845 100826 -19
Branches 13044 13041 -3
=============================================
- Hits 48695 48553 -142
- Misses 46201 46304 +103
- Partials 5949 5969 +20 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Review by github-manager-bot
Summary
Adds a ClientConfig.localOffsetStoreIgnoreCorrupted flag (default false) so that when local offset files are corrupted (e.g., truncated during power loss), the consumer can skip them with a warning instead of crashing with MQClientException.
Findings
- [Info]
LocalFileOffsetStore.java:266— The guard is correctly placed inside thecatch (Exception e)block ofreadLocalOffsetBak(). Returningnullhere causes the caller to fall back to broker-side offset orCONSUME_FROM_TIMESTAMP, which is the desired behavior. - [Info]
ClientConfig.java:108— Defaultfalsepreserves backward compatibility. Good. - [Info] The log message includes
groupNamewhich aids debugging. Consider also logging the file path that was corrupted for faster diagnosis.
Suggestions
- Consider whether
readLocalOffset()(the primary read path, not just the backup) should also respect this flag. Currently onlyreadLocalOffsetBak()is guarded — if the primary file is corrupted, the exception still propagates. - A unit test covering the
localOffsetStoreIgnoreCorrupted=truepath would strengthen this change.
Automated review by github-manager-bot
…et files When ~/.rocketmq_offsets/ files are corrupted (e.g., truncated due to power loss during persistAll()), readLocalOffsetBak() unconditionally throws MQClientException, crashing the consumer application. Fix: Add ClientConfig.localOffsetStoreIgnoreCorrupted (default false). When true, corrupted offset files are silently skipped with a warning log, and the consumer starts with an empty offset table (falling back to CONSUME_FROM_TIMESTAMP or broker-side offset). Default false preserves existing behavior (fully backward compatible). Changes: - ClientConfig: add localOffsetStoreIgnoreCorrupted field + getter/setter - LocalFileOffsetStore.readLocalOffsetBak(): check config before throwing
580f8ca to
39c92da
Compare
RockteMQ-AI
left a comment
There was a problem hiding this comment.
Review by github-manager-bot (Re-review after new commit)
Summary
Re-reviewed after commit at 05:55 UTC. The diff is unchanged from the initial review — the new commit appears to be a rebase on the latest develop.
Status
The previous review findings remain valid:
- Core change is clean:
localOffsetStoreIgnoreCorruptedflag inClientConfigwith properclone/reset/toStringintegration. - Guard correctly placed in
readLocalOffsetBak()catch block, returningnullto trigger fallback. - Default
falsepreserves backward compatibility.
Previous Suggestions (Still Open)
- Primary read path:
readLocalOffset()(not just the backup) could also respect this flag — currently a corrupted primary file still throws. - Unit test: A test covering the
localOffsetStoreIgnoreCorrupted=truepath would strengthen this change.
No new issues found.
Automated review by github-manager-bot
Which Issue(s) This PR Fixes
Fixes #10518
Brief Description
When ~/.rocketmq_offsets/ files are corrupted (e.g., truncated due to power loss during persistAll()), readLocalOffsetBak() unconditionally throws MQClientException, crashing the consumer application.
Fix: Add ClientConfig.localOffsetStoreIgnoreCorrupted (default false). When true, corrupted offset files are silently skipped with a warning log, and the consumer starts with an empty offset table (falling back to CONSUME_FROM_TIMESTAMP or broker-side offset). Default false preserves existing behavior (fully backward compatible).
How Did You Test This Change
Compilation passed. Default false ensures no behavior change for existing users.