Skip to content

[ISSUE #10518] Add client config to ignore corrupted local offset files#10568

Open
wang-jiahua wants to merge 1 commit into
apache:developfrom
wang-jiahua:fix/ignore-corrupted-local-offset
Open

[ISSUE #10518] Add client config to ignore corrupted local offset files#10568
wang-jiahua wants to merge 1 commit into
apache:developfrom
wang-jiahua:fix/ignore-corrupted-local-offset

Conversation

@wang-jiahua

Copy link
Copy Markdown
Contributor

Which Issue(s) This PR Fixes

Fixes #10518

Brief Description

When ~/.rocketmq_offsets/ files are corrupted (e.g., truncated due to power loss during persistAll()), readLocalOffsetBak() unconditionally throws MQClientException, crashing the consumer application.

Fix: Add ClientConfig.localOffsetStoreIgnoreCorrupted (default false). When true, corrupted offset files are silently skipped with a warning log, and the consumer starts with an empty offset table (falling back to CONSUME_FROM_TIMESTAMP or broker-side offset). Default false preserves existing behavior (fully backward compatible).

How Did You Test This Change

Compilation passed. Default false ensures no behavior change for existing users.

Copilot AI review requested due to automatic review settings July 1, 2026 03:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an opt-in client configuration to allow consumers to ignore corrupted local offset backup files (offsets.json.bak) instead of failing startup, addressing the scenario where offset persistence can be non-critical and corruption can occur (e.g., truncation).

Changes:

  • Added ClientConfig.localOffsetStoreIgnoreCorrupted (default false) with accessor methods.
  • Updated LocalFileOffsetStore.readLocalOffsetBak() to optionally skip corrupted .bak content and continue with an empty offset table.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
client/src/main/java/org/apache/rocketmq/client/consumer/store/LocalFileOffsetStore.java Adds opt-in behavior to ignore corrupted local offset backup files during load.
client/src/main/java/org/apache/rocketmq/client/ClientConfig.java Introduces a new client flag controlling corrupted local offset file handling.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread client/src/main/java/org/apache/rocketmq/client/ClientConfig.java
Comment thread client/src/main/java/org/apache/rocketmq/client/ClientConfig.java
@codecov-commenter

codecov-commenter commented Jul 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 33.33333% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.15%. Comparing base (88709c5) to head (39c92da).
⚠️ Report is 2 commits behind head on develop.

Files with missing lines Patch % Lines
.../java/org/apache/rocketmq/client/ClientConfig.java 50.00% 3 Missing ⚠️
...mq/client/consumer/store/LocalFileOffsetStore.java 0.00% 3 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             develop   #10568      +/-   ##
=============================================
- Coverage      48.28%   48.15%   -0.14%     
+ Complexity     13445    13402      -43     
=============================================
  Files           1378     1378              
  Lines         100845   100826      -19     
  Branches       13044    13041       -3     
=============================================
- Hits           48695    48553     -142     
- Misses         46201    46304     +103     
- Partials        5949     5969      +20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by github-manager-bot

Summary

Adds a ClientConfig.localOffsetStoreIgnoreCorrupted flag (default false) so that when local offset files are corrupted (e.g., truncated during power loss), the consumer can skip them with a warning instead of crashing with MQClientException.

Findings

  • [Info] LocalFileOffsetStore.java:266 — The guard is correctly placed inside the catch (Exception e) block of readLocalOffsetBak(). Returning null here causes the caller to fall back to broker-side offset or CONSUME_FROM_TIMESTAMP, which is the desired behavior.
  • [Info] ClientConfig.java:108 — Default false preserves backward compatibility. Good.
  • [Info] The log message includes groupName which aids debugging. Consider also logging the file path that was corrupted for faster diagnosis.

Suggestions

  • Consider whether readLocalOffset() (the primary read path, not just the backup) should also respect this flag. Currently only readLocalOffsetBak() is guarded — if the primary file is corrupted, the exception still propagates.
  • A unit test covering the localOffsetStoreIgnoreCorrupted=true path would strengthen this change.

Automated review by github-manager-bot

…et files

When ~/.rocketmq_offsets/ files are corrupted (e.g., truncated due to
power loss during persistAll()), readLocalOffsetBak() unconditionally
throws MQClientException, crashing the consumer application.

Fix: Add ClientConfig.localOffsetStoreIgnoreCorrupted (default false).
When true, corrupted offset files are silently skipped with a warning
log, and the consumer starts with an empty offset table (falling back
to CONSUME_FROM_TIMESTAMP or broker-side offset). Default false
preserves existing behavior (fully backward compatible).

Changes:
- ClientConfig: add localOffsetStoreIgnoreCorrupted field + getter/setter
- LocalFileOffsetStore.readLocalOffsetBak(): check config before throwing
@wang-jiahua wang-jiahua force-pushed the fix/ignore-corrupted-local-offset branch from 580f8ca to 39c92da Compare July 1, 2026 05:55

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by github-manager-bot (Re-review after new commit)

Summary

Re-reviewed after commit at 05:55 UTC. The diff is unchanged from the initial review — the new commit appears to be a rebase on the latest develop.

Status

The previous review findings remain valid:

  • Core change is clean: localOffsetStoreIgnoreCorrupted flag in ClientConfig with proper clone/reset/toString integration.
  • Guard correctly placed in readLocalOffsetBak() catch block, returning null to trigger fallback.
  • Default false preserves backward compatibility.

Previous Suggestions (Still Open)

  1. Primary read path: readLocalOffset() (not just the backup) could also respect this flag — currently a corrupted primary file still throws.
  2. Unit test: A test covering the localOffsetStoreIgnoreCorrupted=true path would strengthen this change.

No new issues found.


Automated review by github-manager-bot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add client config option to ignore corrupted local offset files instead of throwing exception

4 participants