Skip to content

fix(compactor): Fix flaky TestBlocksCleaner_ShouldRemoveBlocksOutside…#7486

Merged
friedrichg merged 1 commit intocortexproject:masterfrom
yeya24:fix-blocks-cleaner-flaky-test
May 8, 2026
Merged

fix(compactor): Fix flaky TestBlocksCleaner_ShouldRemoveBlocksOutside…#7486
friedrichg merged 1 commit intocortexproject:masterfrom
yeya24:fix-blocks-cleaner-flaky-test

Conversation

@yeya24
Copy link
Copy Markdown
Contributor

@yeya24 yeya24 commented May 6, 2026

…RetentionPeriod

Switch from filesystem bucket to in-memory bucket in this test. The filesystem bucket's Upload is non-atomic (os.Create truncates, then io.Copy writes), creating a race window where the HeartBeat goroutine's write can be observed as an empty file by a concurrent read. The in-memory bucket's Upload is atomic (mutex-protected), eliminating the race.

What this PR does:

Fix flaky test by switching to use an inmemory bucket to eliminate possible race condition when the visit marker is being uploaded.

--- FAIL: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (0.06s)
    blocks_cleaner_test.go:822: 
        	Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:822
        	Error:      	Received unexpected error:
        	            	failed to read cleaner visit marker: visit marker file: markers/cleaner-visit-marker.json, content: , error: unexpected end of JSON input: unmarshal visit marker JSON
        	Test:       	TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

…RetentionPeriod

Switch from filesystem bucket to in-memory bucket in this test. The
filesystem bucket's Upload is non-atomic (os.Create truncates, then
io.Copy writes), creating a race window where the HeartBeat goroutine's
write can be observed as an empty file by a concurrent read. The
in-memory bucket's Upload is atomic (mutex-protected), eliminating the
race.

Signed-off-by: Ben Ye <benye@amazon.com>
Copy link
Copy Markdown
Member

@SungJin1212 SungJin1212 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label May 7, 2026
Copy link
Copy Markdown
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@friedrichg friedrichg merged commit 154f7b3 into cortexproject:master May 8, 2026
68 of 69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/compactor lgtm This PR has been approved by a maintainer size/XS type/flaky-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants