Summary
Automatic lifecycle cleanup skips all staging directories belonging to currentKey, including abandoned directories from a previous crashed worker. This is inherited from #289 and was found during ClickHouse/ai#4173.
Reproduction
- Begin publishing a snapshot for key K and crash after creating
.staging-K-* and its worker manifest.
- Restart preparation for the same unchanged key K.
withDependencySnapshot() obtains K's lock and runs prune with currentKey: K before starting the new operation.- Prune skips every staging entry owned by K without distinguishing the previous abandoned entry from the current publication.
Repeated same-key retries never reclaim the leftovers through automatic cleanup. Staging contents are also excluded from retained-byte accounting, so these artifacts can accumulate beyond the configured snapshot budget. An explicit separate prune or a different-key operation can reclaim them, but the common retry path does not.
Expected fix and regression coverage
Separate protection of the current final snapshot from protection of the exact active staging path. Under the held key lock, clean abandoned same-key staging before starting publication; preserve the currently publishing staging directory. Test crash/restart with the same key and repeated retry, as well as active publication protection.
Summary
Automatic lifecycle cleanup skips all staging directories belonging to
currentKey, including abandoned directories from a previous crashed worker. This is inherited from #289 and was found during ClickHouse/ai#4173.Reproduction
.staging-K-*and its worker manifest.withDependencySnapshot()obtains K's lock and runs prune withcurrentKey: Kbefore starting the new operation.Repeated same-key retries never reclaim the leftovers through automatic cleanup. Staging contents are also excluded from retained-byte accounting, so these artifacts can accumulate beyond the configured snapshot budget. An explicit separate prune or a different-key operation can reclaim them, but the common retry path does not.
Expected fix and regression coverage
Separate protection of the current final snapshot from protection of the exact active staging path. Under the held key lock, clean abandoned same-key staging before starting publication; preserve the currently publishing staging directory. Test crash/restart with the same key and repeated retry, as well as active publication protection.