Skip to content

[fix](fe) Prevent stale recycle candidates from erasing new generations - #67750

Open
wenzhenghu wants to merge 1 commit into
apache:masterfrom
HYDCP:wzh/upstream-fix-recycle-bin-aba
Open

[fix](fe) Prevent stale recycle candidates from erasing new generations#67750
wenzhenghu wants to merge 1 commit into
apache:masterfrom
HYDCP:wzh/upstream-fix-recycle-bin-aba

Conversation

@wenzhenghu

@wenzhenghu wenzhenghu commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #67303

Related PR: HYDCP#103

Problem Summary:

CatalogRecycleBin collects expired IDs under a read lock and erases them later under per-item write locks. During that gap, RECOVER -> DROP can recycle a new object generation with the same ID, allowing the stale candidate to erase the new generation and write an incorrect erase journal.

This change snapshots the recycle-info object and recycle timestamp with each candidate. Under the write lock, it verifies object identity, timestamp, and current expiration before any erase callback, removal, or journal entry. Database, table, and partition paths are all covered.

Static review confirmed that the per-item lock scope remains unchanged and that the change does not alter persisted metadata formats, public APIs, configuration, or upgrade behavior.

Release note

Fix CatalogRecycleBin to preserve newly recycled databases, tables, and partitions when an erase cycle holds a stale expired candidate.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
      • ./run-fe-ut.sh --run org.apache.doris.catalog.CatalogRecycleBinTest (33 tests passed)
      • ./run-fe-ut.sh --run org.apache.doris.catalog.CatalogRecycleBinTest#testExpiredDatabaseSnapshotDoesNotEraseNewGeneration+testExpiredTableSnapshotDoesNotEraseNewGeneration+testExpiredPartitionSnapshotDoesNotEraseNewGeneration (3 tests passed)
      • mvn -pl fe-core checkstyle:check -DskipTests
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. Stale expired candidates are skipped when the same ID has been recycled as a new generation.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: close apache#67303

Related PR: None

Problem Summary: CatalogRecycleBin collected only expired IDs under a read lock. If an object was recovered and recycled with the same ID before the erase worker reacquired the write lock, the stale candidate could erase the newly recycled generation. Snapshot the recycle info and timestamp, then validate identity, timestamp, and expiration under the write lock before any erase callback, removal, or journal entry.

### Release note

Fix CatalogRecycleBin to preserve newly recycled databases, tables, and partitions when an erase cycle holds a stale expired candidate.

### Check List (For Author)

- Test: Unit Test
    - Added deterministic FE unit tests for database, table, and partition recover/recycle races.
    - `./run-fe-ut.sh --run org.apache.doris.catalog.CatalogRecycleBinTest`
    - `mvn -pl fe-core checkstyle:check -DskipTests`
- Behavior changed: Yes. Stale expired candidates are skipped when the same ID has been recycled as a new generation.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wenzhenghu

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] CatalogRecycleBin microbatch erase has an ABA race and retains a long-held lock in DB cascade cleanup

2 participants