fix: ListObjects should return NoSuchBucket for prefix on missing bucket

[ycjlin]

Contribution licensing

This maintainer-submitted revision is licensed inbound=outbound under the repository AGPL-3.0-or-later terms and carries a matching DCO sign-off. It preserves Jason Lin as a co-author of the original contribution.

Problem

ListObjects on a missing bucket can return an empty 200 instead of NoSuchBucket when listPath exits before consulting storage. The affected shortcuts are:

• a prefix beginning with /;
• max-keys=0;
• a marker outside the requested prefix.

The regression came from upstream 80ca12008 / minio#18917, which removed generic bucket-existence checks to avoid cluster-wide GetBucketInfo fan-out. Normal listing still discovers a missing bucket through storage, but these shortcuts never reach that path.

Fixes #32.

Implementation

Verify bucket existence only immediately before the three storage-bypassing EOF returns. An existing bucket keeps the original empty-list result; a missing bucket reaches the existing 404 NoSuchBucket mapping. The normal listing hot path remains unchanged.

This deliberately does not restore the generic existence check or add a cache. GetBucketInfo fans out to peers and disks, so only shortcut traffic pays that cost.

Tests

The updated revision covers ListObjects, ListObjectsV2, and ListObjectVersions for:

• slash-prefixed prefix;
• zero limit;
• marker outside prefix;
• a regular storage-backed prefix as a control.

HTTP tests verify signed V1, V2, and version-list requests return status 404 with XML code NoSuchBucket.

Local verification on exact head e9c5340be:

• go test ./cmd -count=1;
• focused regression and race tests;
• related existing listing tests;
• CGO_ENABLED=0 go build ./...;
• go vet ./...;
• gofmt and diff checks.

A separate local Claude Code Fable Max adversarial review returned GO with no mandatory pre-merge change.

Compatibility and performance

This restores AWS S3 behavior and the behavior before 80ca12008. Clients that relied on the incorrect empty 200 will now receive 404. Shortcut calls are more expensive because they perform GetBucketInfo; normal listings do not. Large-cluster shortcut frequency and tail latency remain post-merge observability items, not reasons to reintroduce a generic check.

Checklist

• Fixes a regression (80ca12008)
• Unit and HTTP tests added
• DCO sign-off present
• Local build, vet, tests, race, and formatting checks passed
• Design record prepared in pgsty/silo.pgsty.com
• Remote CI green on the exact PR head

[ycjlin]

Thanks @cbornet for the prompt review.
cc: @Vonng for merging.

[Vonng]

Thanks for the PR. I looked into the history behind this behavior.

The relevant upstream change is 80ca12008 / minio#18917, which removed the unconditional checkBucketExist call from several object API argument checks. The motivation was performance: GetBucketInfo() can fan out across peers and becomes costly on large distributed deployments, especially with 100+ nodes.

Because of that, I am concerned about reintroducing bucket existence checks too broadly. Moving the check back into the generic ListObjects argument path would effectively undo the upstream optimization and could bring back the same scalability issue.

This PR is much narrower than a full revert, but it is still a user-visible behavior change: some ListObjects calls against a non-existent bucket would change from an empty result to NoSuchBucket. Given that this fork is intended to stay close to upstream and is mainly maintained for binary supply-chain/security purposes, my current preference is to keep this behavior aligned with upstream for non-security fixes.

[cbornet]

I understand your concern but:

• returning a NoSuchBucket is more correct.
• the change from 80ca12008 was already a user-visible change. We would just re-establish it to the behavior it used to have. In my case I want to upgrade from a version prior to the 80ca12008 and I can't do it precisely because of that behavior change.
• The definition of silo/minio is MinIO is a Kubernetes-native S3-compatible object storage solution and S3 returns NoSuchBucket. A lot of (most?) people use Minio as a S3 simulation during unit/integration tests so having the same behavior as S3 is very important.

[Vonng]

Thanks for the fix and the accompanying test. Review is queued right after the RELEASE.2026-08-06 cut: server-visible behavior was deliberately frozen while that release went through acceptance, and this touches the metacache listing path, which deserves an unhurried review rather than a release-eve merge. Follow-up here shortly.

[Vonng]

Approved on exact head e9c5340 after local implementation review, independent Claude Fable Max adversarial review, and green DCO, VulnCheck, and all six Go CI jobs. The compatibility fix is intentionally limited to the three storage-bypassing list shortcuts; the normal listing hot path remains unchanged. The cluster-wide GetBucketInfo cost on shortcut traffic is documented and accepted.