S3 backup repository: stop writing and depending on directory marker objects

[elangelo]

Supersedes #4182 (closed as stale, and it can't be reopened after the branch was rebased). See that PR for the earlier review discussion.

Description

The S3 backup repository simulated directories with zero-byte "directory marker" objects (keys ending in /, content type application/x-directory), and relied on them when restoring. Tools that copy buckets, such as aws s3 sync, only copy the real files and drop those markers. A backup copied that way could no longer be restored: the configset directory, among others, was reported as missing.

Solution

Following the review on #4182, the S3 repository no longer uses directory markers at all, rather than adding a fallback for when they are missing:

• S3StorageClient.createDirectory() is a no-op. On S3 a directory is just a key prefix that has objects under it.
• exists() on a directory path and isDirectory() use a single prefix listing (maxKeys=1). This finds markerless directories as well as marker objects left by older Solr versions, so existing backups can still be restored. deleteDirectory() also removes those old markers.
• Writing a file no longer checks that its parent "directory" exists, and reading a file's length takes a single HEAD request again. Both mean fewer S3 requests.
• Core backup/restore code keeps calling createDirectory()/exists(). That keeps LocalFileSystemRepository, which needs real parent directories, and the other repositories working as before.
• BackupRepository gets supportsEmptyDirectories(), true by default and false for S3. A collection backup on such a repository accepts a location that does not exist yet, as it comes into existence with the first file of the backup; before, a backup to a new S3 prefix was refused until something was stored there. The location is still checked, so an unreachable bucket fails up front. Restore, list and delete still require the location to exist.

On S3 an empty directory can't exist any more. The shared AbstractBackupRepositoryTest contract assumed it could; it now asks the repository through supportsEmptyDirectories(), so S3 runs the shared tests instead of copying and overriding them.

Copilot (#4182) and Claude Code were used to develop this change.

Tests

All new tests run against S3Mock instead of Mockito mocks:

• S3IncrementalBackupTest#testRestoreBackupCopiedWithoutDirectoryMarkers is the end-to-end regression test. It backs up a collection and checks the backup contains no marker objects. It then copies the backup file by file to another location, as aws s3 sync would, and restores it under a new configset name so the configset is uploaded from the backup. Finally it checks the configset and the document count.
• S3IncrementalBackupTest#testBackupToLocationThatDoesNotExistYet backs up to a prefix that holds nothing yet, restores it, and checks that restoring from a missing location still fails.
• S3BackupRepositoryTest#testLegacyDirectoryMarkers checks that a backup containing old-style markers can still be checked for existence, listed, read and fully deleted.
• S3PathsTest, S3ReadWriteTest and S3BackupRepositoryTest are updated for markerless directories.

Ran :solr:modules:s3-repository:test -Ptests.nightly=true; all pass except S3InstallShardTest#testInstallSkipsWriteLockFile, which fails the same way on current main and is unrelated to this change. I also ran LocalFileSystemRepositoryTest and GCSBackupRepositoryTest to cover the shared contract test change, BackupRestoreApiErrorConditionsTest and TestLocalFSCloudBackupRestore to check that a local file system location must still exist, and check -x test on the changed modules.

Checklist

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide
• I have added a changelog entry for my change

🤖 Generated with Claude Code