feat(s3): bring koc s3 to parity with s5cmd

[ftarasenko]

Closes the gap between koc s3 and s5cmd, measured against its actual command surface (command/*.go in that repo) rather than its README.

Before this, koc s3 could list, show, download and upload one object at a time. Nothing could create a bucket, remove anything, move data larger than 5 GiB, or work on more than one object per invocation.

What lands

Bucket and object lifecycle — bucket create/delete/show/set, object delete, du. A bucket's whole lifecycle no longer needs s5cmd or s3cmd alongside koc.

Transfers

• Uploads are multipart past --part-size (16 MiB default), --concurrency parts at once, so the 5 GiB single-PUT ceiling is gone. A failure aborts the upload, so a half-written object never becomes visible.
• upload - reads standard input: mysqldump | gzip | koc s3 upload - db-backups/nightly.sql.gz. Multipart is what makes this possible — SigV4 must hash a single-PUT body before sending it, and a pipe cannot be rewound.
• --recursive (or a wildcard reference) on download, upload, copy, move and object delete, --concurrency objects at a time, with --include/--exclude globs and --dry-run.

New verbs — copy/move (server-side, so the bytes never travel through koc), presign (local signing, no request), sync (local↔S3 and S3↔S3, transferring only what differs).

Other — retry with exponential backoff (--s3-retries), --s3-anonymous, batched DeleteObjects (1000 keys per request instead of one DELETE each), object list --delimiter /, --human sizes, and object versioning (bucket show/set --versioning, --all-versions, --version-id).

Deliberately not implemented: s5cmd's select (S3 Select SQL — AWS-only, absent from Garage) and run (a batch-command file, which exists to feed s5cmd's worker pool).

Cost

Zero new modules — the graph is still 9, and vendor/, go.mod and go.sum are untouched. No new stdlib import either. The binary grew 16,490,658 → 16,744,610 bytes (+253,952, +1.54%), of which sync alone was +40,960 (+0.245%).

Design decisions worth reviewing

• sync requires the s3:// prefix, alone in this group. Every other verb takes a bare <bucket>/<key> because it has one subject; sync has two, and mistaking which side is local is how a --delete removes the wrong one.
• sync compares size and Last-Modified. S3 keeps no record of a source file's mtime, and — decisively — a listing would not return one if it did, so x-amz-meta-mtime would cost a HEAD per object. Last-Modified is correct in the steady state: an upload lands after the file was written, a download after the object was, so neither direction re-transfers what it just moved. The one seam is a round trip, where a restored tree carries the restore time and syncing it back re-uploads once before settling; documented, with --size-only as the way out. Timestamps compare truncated to the second, or a nanosecond local mtime makes every file look newer than its own copy.
• --delete is refused when the source turned out to be empty unless --force is also given. A mistyped source that lists nothing would erase the destination, and that is not recoverable by re-running the command. Entries --include/--exclude leave out are outside the sync and are never deleted.
• A recursive download refuses a key that would escape the destination. ../../etc/cron.d/x is a legal S3 key; writing it where it says would let anyone who can write to the bucket write anywhere koc can.
• Recursive download skips existing files so an interrupted restore resumes by re-running; the single-object form still errors without --force, keeping the existing "don't clobber a backup" behaviour.
• --concurrency is not squared. Under --recursive the pool's workers are objects and each object's multipart then goes out a part at a time, so --concurrency 8 does not mean 64 live part buffers.
• A copy and a multipart completion can both be refused inside a 200 response. Both paths check the body for an <Error> document rather than decoding only the fields they want and reporting success.
• Exact bytes stay the default everywhere; --human is opt-in, and the formatter went into internal/output rather than inline.
• request.body is now an io.ReadSeeker so a retry can replay it, rewound to the caller's offset rather than to byte zero. ListObjects/ListObjectsFunc take a ListOptions struct instead of three positional arguments.

Verification

Presigned URLs are asserted against golden signatures from an independent SigV4 query-signing implementation written from the AWS specification. A URL is only useful if a server accepts it, so agreeing with a second implementation is the property worth testing — a test that re-derived the value with these same functions would pass however wrong both were.

The whole surface was then exercised end to end against a mock endpoint that verifies SigV4 itself (header and query) and speaks the real multipart protocol: stdin upload; a 12 MiB three-part upload with part ordering and reassembly asserted; recursive download/upload with skip-on-exists; wildcard selection; server-side copy and move; batch delete with a checked Content-MD5; bucket show; du --group --human; a presigned URL fetched with curl (and a tampered one rejected); and all three sync directions including --delete both ways, its empty-source refusal, --dry-run, --include scoping both sides, and the documented round-trip seam re-uploading once and then settling. Every request in those runs verified its signature; the only failures logged were the deliberate ones.

Offline gate on each commit: gofmt clean, go vet clean, golangci-lint v2.13.2 → 0 issues, go test ./... green, -race green, the -mod=vendor GOPROXY=off static build, and all six cross-build targets.

docs/coverage.md was re-derived mechanically rather than edited by hand — 565 leaf commands, 50 koc-native, all three arithmetic identities hold.

Note for the maintainer

gofmt from go1.27.1 disagrees with the go1.27.0-built golangci-lint this repo pins, and wants to reformat three files this PR does not touch (internal/auth/provider_test.go, internal/cli/baremetal/allocation.go, internal/cli/network/qos.go) in exactly the multi-value-composite-literal-return shape AGENTS.md warns about. New code here avoids that construct entirely, so the lint gate is clean — but make fmt under 1.27.1 would revert those three and the lint job would then reject them. AGENTS.md's "gofmt here means Go 1.27's" may need a patch-level qualifier.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TQ2Xt7Epc3n8XAXiAgPXiC

---

Generated by Claude Code