Skip to content

fix: delete the previous content when a file is replaced - #419

Open
justadityaraj wants to merge 1 commit into
Forceu:masterfrom
justadityaraj:fix/replace-deletes-previous-content
Open

fix: delete the previous content when a file is replaced#419
justadityaraj wants to merge 1 commit into
Forceu:masterfrom
justadityaraj:fix/replace-deletes-previous-content

Conversation

@justadityaraj

Copy link
Copy Markdown

Description

Closes #407.

ReplaceFile rewrites the existing metadata entry in place, pointing it at the new content's SHA1, AwsBucket and size. Whatever the file used to hold is then referenced by no entry at all.

CleanUp frees a source by walking metadata: for each entry it considers expired or missing it checks whether any other entry shares the same SHA1, and only then removes the stored data. Since nothing points at the replaced content any more, CleanUp never visits it, so it is never reclaimed. That is why the storage stays occupied whether or not deleteNewFile is set: deleteNewFile only expires the entry holding the new content.

The fix keeps a copy of the entry before it is rewritten and deletes that source afterwards, unless another entry still shares the same SHA1 (a duplicate, or a replace with identical content).

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactor / Chore

Technical Details

  • Database changes: No
  • Storage backend affected: Both. The deletion goes through the existing deleteSource, which already routes to aws.DeleteObject for non-local files and keeps the old AwsBucket from the copied entry.
  • Usage of AI: Yes. AI assisted with tracing the ReplaceFile -> CleanUp path and drafting the test. I confirmed the root cause against the code, decided the approach, and ran the suite myself.

How Has This Been Tested?

  • Unit Tests: new TestReplaceFileDeletesPreviousContent writes both sources to the data dir, replaces one with the other, and asserts the previous source is gone while the new one remains. It fails on master (Assertion failed, got: false, want: true) and passes with the change. TestReplaceFile still passes.
  • Environment: Windows, go test ./internal/storage/... --tags=test,awsmock.
  • Four tests fail identically before and after this change on this host: TestNewFile, TestGetFileByChunkId, TestLocalStorageDriver_Init and TestGetDataPath. They look like path-separator assumptions rather than anything related to this change.
  • go vet --tags=test,awsmock ./internal/storage/ and gofmt -l clean.

Checklist

  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.

ReplaceFile rewrites the metadata entry in place to point at the new
content, which leaves the data the file used to hold with no entry
referencing it. CleanUp walks metadata and only frees a source it can
still reach from an entry, so that data is never reclaimed, with or
without deleteNewFile.

Capture the entry before it is rewritten and delete its source, unless
another entry still shares the same SHA1.

Closes Forceu#407
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] API replace does not delete original file immediately.

1 participant