feat: Add Garbage Collection (GC) and MaxArenasToKeep feature

[eeliu]

Description

This pull request introduces an automatic and manual Garbage Collection (GC) mechanism to bigqueue. By default, bigqueue retains all arena files indefinitely, which can lead to storage exhaustion for long-running queues. This feature enables users to periodically or automatically clean up consumed arena files.

Key Features and Implementation Details

1. New Configuration - SetMaxArenasToKeep(n): Users can configure the queue to keep a maximum of n consumed arenas. Expired arenas before this threshold are deleted from the disk.
2. Manual GC Trigger - GC(): Added an explicit GC() method allowing users to manually trigger disk cleanup (e.g., during off-peak hours).
3. Data Structure Refactoring: Modified the internal arenas collection in arenaManager from a []*mmap.File (slice) to a map[int]*mmap.File. This is necessary to support non-contiguous Arena IDs that arise when old files are deleted from the disk.
4. Metadata Preservation: Adjusted the metadata synchronization logic to ensure that if GC removes older head arenas, the global head is advanced correctly and seamlessly persisted to disk.

GC Workflow

flowchart TD
    A[Trigger GC] --> B[Gather Consumer Heads]
    B --> C[Calculate minHeadAid = min of all consumers]
    C --> D{Is minHeadAid valid?}
    D -- Yes --> E[Update Global Head to minHeadAid]
    D -- No --> Z[Exit GC]
    E --> F[Calculate limitAid = minHeadAid - maxArenasToKeep]
    F --> G{limitAid > 0?}
    G -- Yes --> H[Iterate aid from 0 to limitAid-1]
    H --> I[Unmap arena from memory]
    I --> J[Delete .dat file from disk]
    J --> K[Remove from in-memory arena map]
    K --> L[Repeat for next expired arena]
    L --> Z
    G -- No --> Z

Loading

Test Cases Introduced

• gc_test.go: Contains basic functionality tests verifying that configuring SetMaxArenasToKeep correctly cleans up the anticipated arena files upon consumption.
• gc_concurrency_test.go: Highly concurrent stress tests running Enqueue, Dequeue, and GC precisely at the same time to ensure no race conditions arise during the memory unmap or file deletion stages.
• crash_recovery_test.go: Tests the resiliency of the queue during unexpected closures. It simulates a crashed state while an ongoing GC is only partially completed, verifying that the queue can restore itself correctly upon the next boot.
• bigqueue_test.go (Updates): Validation checks ensuring that negative numbers for maxArenasToKeep return the appropriate initialization errors.

All tests are passing cleanly with expected coverage. Please let me know if there are any aspects of the implementation you would like me to adjust.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR introduces configurable garbage collection (GC) for arena files in bigqueue, adds a public GC() entrypoint, and expands documentation and tests to validate cleanup, concurrency, and crash recovery behaviors.

Changes:

• Added SetMaxArenasToKeep configuration and metadata head updating to support arena file garbage collection.
• Implemented arena deletion logic inside arenaManager and exposed MmapQueue.GC() as a public API.
• Added extensive GC + concurrency + crash-recovery tests and updated docs/README examples to the NewMmapQueue API.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
metadata.go	Re-enables putHead so GC can persist updated global head in metadata.
config.go	Adds maxArenasToKeep config, option setter, and validation error.
bigqueue.go	Exposes MmapQueue.GC() to trigger arena cleanup with queue locking.
arenamanager.go	Refactors arena tracking (slice→map) and implements GC deletion logic + head updates.
gc_test.go	Adds multi-scenario tests validating arena deletion and consumer-head semantics.
gc_concurrency_test.go	Adds concurrent producer/consumer test with periodic GC.
crash_recovery_test.go	Adds multi-process crash recovery tests for enqueue, dequeue, and torn GC state.
bigqueue_test.go	Adds unit test for negative SetMaxArenasToKeep validation.
doc.go	Updates examples to NewMmapQueue and documents GC usage.
README.md	Documents SetMaxArenasToKeep and manual GC() usage; updates examples to NewMmapQueue.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mangalaman93]

@eeliu please rebase on latest main so that the tests can pass.

[mangalaman93]

@eeliu help me understand that the comparison between using an array vs map? I also do not understand why we may end up deleting arenas in non-contiguous manner?

[eeliu]

@mangalaman93

Glad to do that, please give me sometime.

[eeliu]

The switch from an Array to a Map was implemented to support data reclamation (GC) and multi-consumer modes.

1. Sparse Management: Maps do not require continuous indexing. Old Arenas can be removed via a simple delete, avoiding the "index holes" or slice-shifting operations necessary with arrays.
2. GC Compatibility: Since deletion is driven by the slowest consumer, the Map structure easily manages active blocks across large ID spans without creating invalid placeholders.
3. Logical Simplification: This refactor deprecates the cumbersome maintenance of baseAid and slice offset logic, reducing the risk of index overflows and state corruption.
4. Performance Balance: The memory eviction strategy continues to prioritize the protection of Head and Tail Arenas, ensuring core performance for sequential read/write remains unaffected.

Conclusion: This trade-off accepts a negligible lookup overhead in exchange for storage management flexibility and improved system scalability.

[mangalaman93]

@eeliu this looks like an AI generated answer. What I am looking to understand is that if we use an array, we can ensure that they are in sequence. We can always use some sort of a vector structure or a ring structure to ensure that we can reuse the memory. The benefit of using an array is that, the arenas are contiguous which is how they are in memory too.

I am wondering whether there can be a case where we can have say arena 2 & 4 but we may have deleted arena 3.

[eeliu]

@mangalaman93

Got it.

I am wondering whether there can be a case where we can have say arena 2 & 4 but we may have deleted arena 3.

gc alg searched the smallest id from arena, so can't delete 3 with 2 exist.

IMP, bigqueue works like a cache(disk), if user needs to store data , try to add a new consumer.

[mangalaman93]

@mangalaman93

Got it.

I am wondering whether there can be a case where we can have say arena 2 & 4 but we may have deleted arena 3.

gc alg searched the smallest id from arena, so can't delete 3 with 2 exist.

IMP, bigqueue works like a cache(disk), if user needs to store data , try to add a new consumer.

That makes sense. That is why I am thinking that if we use an array, it would be easy to find the lowest arena in O(1). We can use a ring buffer instead. That would be a better fit here.

[mangalaman93]

@eeliu the CI is not happy, please look into it and check my last comment when you can.

[eeliu]

sorry for later response, busy these days.

As you mentioned ring buffer, it works perfect under a fixed size of arena, but under my case ( cached size range 10MB ～ 1TB ), not a good choice.

[mangalaman93]

sorry for later response, busy these days.

As you mentioned ring buffer, it works perfect under a fixed size of arena, but under my case ( cached size range 10MB ～ 1TB ), not a good choice.

could you explain a bit more when you get some time? Thanks

[eeliu]

in our case, cached file size is 126MB 99% time, only when back-ends down it may reach 1TB.

So map would be better for me.

[mangalaman93]

in our case, cached file size is 126MB 99% time, only when back-ends down it may reach 1TB.

So map would be better for me.

But map would consume a lot of memory whereas array would consume a lot less memory.