Harden per-bucket CORS replication, recovery, and compatibility after #71

[Vonng]

Context

PR #71 adds the standard S3 per-bucket CORS control plane and enforcement path. The core single-site implementation has been accepted: PutBucketCors, GetBucketCors, and DeleteBucketCors work with a standard minio-go client; matching and rejected preflights behave as expected; configured response headers are applied; and buckets without a CORS configuration retain the existing global fallback.

The maintainer is accepting the contribution and taking ownership of the remaining integration and compatibility hardening in this issue. This issue is a release gate for shipping per-bucket CORS; merging #71 does not by itself make the feature release-ready.

CORS is a browser response policy, not an IAM authorization mechanism. The replication bugs below do not bypass S3 permissions, but they can leave an origin enabled on one site after an administrator intended to revoke it.

Release-blocking correctness work

1. Preserve CORS tombstones and source ordering across site replication

The current peer handler and heal path do not preserve last-writer-wins semantics across PUT and DELETE:

• SiteReplicationMetaInfo only exports CorsConfigUpdatedAt when CorsConfigXML is non-empty. A DELETE therefore loses its tombstone timestamp in the status/heal snapshot.
• PeerBucketCorsConfigHandler checks staleness through GetCorsConfig. Once the local config is deleted, that getter returns not-found and the newer deletion timestamp is unavailable, so an older PUT can resurrect the config.
• Peer PUT/DELETE calls use BucketMetadataSys.Update/Delete, which generate a new local timestamp instead of persisting the source UpdatedAt. Delayed or reordered events can therefore be accepted or rejected using arrival time rather than source order.
• When one site misses a DELETE, heal can currently select the remaining old non-nil config as the newest state and restore it to the site where it was deleted.

Required behavior:

• Default CorsConfigUpdatedAt consistently for metadata that predates this field.
• Export CorsConfigUpdatedAt even when CorsConfig is nil, so a deletion remains an observable tombstone.
• Compare incoming events against the raw stored CORS timestamp regardless of whether a live config exists.
• Persist the incoming source UpdatedAt for both PUT and DELETE instead of replacing it with local arrival time.
• Make duplicate, delayed, and reordered CORS PUT/DELETE events idempotent and last-writer-wins.
• Make heal choose and propagate a newer nil tombstone over an older non-nil config.

2. Add tests against the real apply/status/heal seams

The tests added in #71 cover JSON/base64 transport and pointer equality, but not metadata application or convergence. Add regression coverage for:

• A newer peer DELETE is applied after a delayed older peer PUT.
• An older peer PUT cannot resurrect a config after a newer local or peer DELETE.
• A CORS deletion timestamp survives SiteReplicationMetaInfo with CorsConfig == nil.
• Heal removes a stale peer config when the newest state is a tombstone.
• Normal peer PUT, replacement PUT, and DELETE preserve the exact source timestamps.
• Initial site sync and retry are idempotent.
• At least one multi-site or faithful peer-handler integration test exercises the full dispatch-to-metadata path.

3. Do not fail open on metadata errors

The middleware should use the global CORS fallback only when the bucket genuinely has no per-bucket configuration. A parse, cache, or operational metadata error must not silently widen the bucket to the global policy.

• Distinguish no CORS config from metadata/parse failures.
• Define and test the safe response behavior for a configured bucket whose CORS metadata cannot be read.

Compatibility and operational cleanup

4. Complete the S3 preflight response

• Return configured Access-Control-Expose-Headers on a successful OPTIONS response.
• Keep Vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers coverage.
• Add response-level tests for allowed headers, exposed headers, max age, and cache variation.

5. Correct site-replication status accounting

TotalCorsConfigCount currently uses the cumulative corsCfgCount > 0 condition while iterating sites. Once an earlier site has a config, a later site without one can be counted as configured.

• Increment each site's total from that site's s.CorsConfig != nil state.
• Add asymmetric-site tests: config on first site only, last site only, all sites, and no sites.
• Audit the neighboring tag/SSE/version counters for the same inherited cumulative-count pattern; fix them in a separate focused commit or open dedicated issues if the scope is broader.

6. Finish validation edge cases

• Reject empty AllowedOrigin values rather than only checking that an element exists.
• Decide and test whether methods must use the canonical uppercase S3 values.
• Ensure only the documented * wildcard semantics are accepted; do not inherit ? wildcard behavior from a generic matcher.
• Treat the 255-character rule ID limit as characters rather than bytes, or document the deliberate byte limit.
• Add malformed checksum, missing checksum, oversized XML, trailing XML, and invalid-rule HTTP tests.

7. Document upgrade, fallback, and rollback semantics accurately

CORS data is stored in the bucket metadata record under the logical cors.xml selector; it is not an independent dormant file that is guaranteed to survive all downgrade writes.

• Document that an older binary does not enforce the new config.
• Document that an older binary rewriting bucket metadata may drop unknown CORS fields.
• Document that a bucket-level config overrides, rather than merges with, the global CORS policy; DELETE restores the global fallback.
• Add/update public operator documentation in pgsty/silo.pgsty.com as a separate repository deliverable.

Inherited site-replication pattern audit

The CORS implementation mirrors existing policy/tag/SSE/quota handlers. Some of the timestamp and cumulative-counter behavior may therefore predate #71.

• Audit existing bucket-metadata peer handlers for the same tombstone/source-timestamp problem.
• Prefer a small shared source-timestamp-preserving helper if it reduces duplicated correctness logic without broad refactoring.
• If other metadata types are demonstrably affected, open focused issues with reproductions instead of silently expanding this change without bounds.

Verification and release gate

• go build ./...
• Focused CORS and site-replication tests
• Focused race tests
• Full cmd package tests
• golangci-lint with the repository-pinned version
• Generated-file and compatibility checks
• Standard minio-go end-to-end PUT/GET/DELETE and preflight smoke test
• GitHub CI green on the follow-up PR and on main
• No release tag or Docker publication containing feat: per-bucket CORS configuration (S3 ?cors) with enforcement #71 until the release-blocking items above are complete

Out of scope

• Silo Console UI for editing bucket CORS is a separate follow-up.
• A broad redesign of site replication is not required unless the audit produces a concrete shared bug and a narrowly reviewable fix.

Acceptance criteria

This issue can close when per-bucket CORS converges correctly across normal delivery, retry, reordering, missed DELETE, and heal; errors cannot widen a configured bucket to the global policy; the protocol/status cleanup is tested; documentation describes the operational boundary; and all local and remote release gates are green.

[Vonng]

Implementation plan after merging #71

The merged tree is now checked out on codex/issue-75-cors-hardening. Keep the work in small reviewable commits and preserve the release gate in this issue.

Phase 1 — reproduce first, then close the convergence hole

1. Add failing object-layer tests for the three confirmed cases before changing production code:
   • newer DELETE after delayed PUT is not ignored;
   • stale PUT cannot resurrect a newer tombstone;
   • SiteReplicationMetaInfo retains the timestamp when CorsConfig == nil.
2. Add a narrow CORS metadata apply helper that:
   • reads raw bucket metadata, including the timestamp when the live config is nil;
   • rejects zero/older/equal conflicting events consistently;
   • decodes and validates a non-nil payload;
   • writes the source timestamp exactly for PUT and DELETE;
   • saves through BucketMetadataSys so cache and notifications remain coherent.
3. Use the same helper from peer apply and the local branch of heal. Do not use Update/Delete where they replace source time with arrival time.
4. Always export CorsConfigUpdatedAt; default the field to bucket creation time for pre-feature metadata.
5. Add a heal regression where one site has an older config and another has a newer nil tombstone. The tombstone must win and converge on every site.

Suggested first commit boundary: fix: preserve CORS replication tombstones and source order with all convergence tests included.

Phase 2 — make middleware failure behavior explicit

1. Treat only errConfigNotFound as permission to use the global fallback.
2. For an operational/parse error on a cross-origin request, fail closed: do not apply the wider global CORS policy. Define the HTTP behavior separately for preflight and actual requests and test both.
3. Avoid a per-bucket metadata lookup for requests without an Origin header if this can be done without changing existing global middleware semantics.
4. Return configured expose headers on successful preflight responses.

Suggested second commit boundary: fix: fail closed on bucket CORS metadata errors.

Phase 3 — protocol validation and status correctness

1. Replace generic wildcard matching with the documented S3 single-* behavior; do not treat ? as a wildcard.
2. Reject empty origins, decide canonical uppercase method handling, count rule IDs by characters, and reject trailing XML.
3. Add HTTP-level checksum and malformed-document cases.
4. Correct TotalCorsConfigCount using the current site's config state and add asymmetric-site tests.
5. Run focused reproductions against neighboring tag/SSE/version counters. If they fail, open dedicated issues before changing those types; do not fold a broad site-replication rewrite into this branch.

Suggested third commit boundary: fix: complete bucket CORS protocol and status validation.

Phase 4 — documentation and release proof

1. Document override/fallback and downgrade metadata-loss semantics in the owning repositories.
2. Run focused normal/race tests, full cmd, pinned lint, generated checks, and the standard-client end-to-end test.
3. Push the branch, open a focused PR referencing this issue, and require GitHub CI green before closing Harden per-bucket CORS replication, recovery, and compatibility after #71 #75 or publishing a release.

Suggested final commit boundary: documentation only, kept separate from server behavior.

Scope rule

The first PR from this branch should make CORS itself release-safe. Any confirmed pre-existing bug shared by policy/tag/SSE/quota should be recorded with its own reproduction and issue unless a tiny shared helper is clearly safer than duplicated fixes.

[Vonng]

Post-merge baseline is green on main at e4e3007da6d7d1198a6a050e34f84566d40a9654: Go CI run 33136344507 completed successfully across cross-compile, internal/cmd tests, format/build/vet, lint/generated files, and race/S3 Select; VulnCheck run 33136344570 also passed. This records the accepted baseline only; the release gate in this issue remains open until the follow-up fixes and their own CI are complete.

[Vonng]

Local implementation and adversarial review status

The #75 repair is implemented in the isolated codex/issue-75-cors-hardening worktree, but remains uncommitted and unpushed.

Implemented:

• source-timestamp-preserving CORS peer PUT/DELETE with nil tombstones and strict-After ordering;
• missing-metadata rejection and pre-feature timestamp defaults;
• unconditional tombstone timestamp export and heal through the same apply path;
• heal selection that excludes creation-time defaults before seeding latest;
• fail-closed metadata errors without a new S3 500 contract;
• preflight ExposeHeaders and complete success/rejection Vary behavior;
• literal * versus patterned-origin credential semantics, including mixed-origin ordering;
• CORS per-site total, empty-origin/? validation, and Content-MD5 negative coverage;
• a checksum verification fix: remove the exact-length LimitReader that prevented the shared checker from surfacing BadDigest.

Verification on the final diff:

• focused normal tests: pass;
• full internal/bucket/cors race: pass;
• focused cmd CORS race: pass;
• full cmd with CI tags: pass (194.150s final run);
• go build ./...: pass;
• pinned golangci-lint v2.13.1: 0 issues;
• generated-file checks: pass;
• compatibility/rebrand/delivery checks: pass with unchanged manifest;
• real minio-go PUT/GET/DELETE, explicit/literal-wildcard responses, credentials, ExposeHeaders, and rejected-preflight Vary: pass.

Review:

• self-adversarial review found and fixed raw-metadata defaulting and log-once key scope;
• Fable 5 Max initial code review: GO WITH FIXES (no P0/P1);
• findings fixed: parsed-cache serving assertion, unmatched-actual Vary: Origin, and heal's creation-default seed corner;
• same-session Fable 5 Max re-review: GO, with no P0/P1/P2 findings.

Documentation:

• bilingual public design record is implemented and fully built in the separate codex/issue-75-cors-design worktree;
• warning-fatal Hugo build and 471,722 rendered internal-link checks pass;
• documentation remains uncommitted/unpushed and is still a separate release gate.

Historical findings are tracked separately in #76 and #77. Nothing from those issues was folded into this CORS branch.

Keep #75 open until the server and documentation changes are committed, pushed, reviewed, merged, and their remote/main CI gates are green.

[Vonng]

B2+B3 final solution frozen locally — verification scheduled next

The release-hardening work has moved beyond the earlier “uncommitted/unpushed” status recorded above.

Server state

• B2 convergence implementation: 724f8703d, submitted as PR fix: complete per-bucket CORS release hardening #80.
• PR fix: complete per-bucket CORS release hardening #80 currently has 8/8 green checks: DCO, VulnCheck, format/build/vet, lint/generated, race/S3 Select, cmd, internal, and cross-compile.
• The standalone B3 strict-wire implementation (ae879f6cc) has now been integrated with B2 in local signed commit 0eebc928f on codex/issue-75-cors-hardening.
• The integration preserves the C-prime register (zero baseline, durable tombstone, exact source time, deterministic equal-time conflicts, distributed CORS lock, full-state status/heal) and adds strict XML/wire validation, Unicode ID counting, exact S3 methods, checksum classification, MaxAge presence, wildcard credentials, and Origin: null handling.
• A combined read-only Claude Code Opus 5 Max review found one conflict-helper typo and one legacy-invalid metadata recovery risk. Both are fixed: legacy-invalid CORS now leaves other bucket metadata readable, fails browser behavior closed, rejects new invalid saves, and can be repaired by a valid CORS PUT or DELETE.
• Focused combined parser, middleware, replication, namespace, and legacy-repair tests pass.

Design and documentation

• The internal implementation contract is tracked in docs/site-replication/CORS-LWW-DESIGN.md.
• Public EN/ZH replication and wire-contract design records are updated locally on codex/issue-75-cors-design, rebased to the latest documentation main, with DCO sign-off.
• No documentation is pushed or deployed yet.

Deliberately deferred until the solution was frozen

The following are now ready to be scheduled, but have not been claimed complete:

1. full combined normal/race/build/vet/lint/compatibility/client verification;
2. final exact-commit real two-site PUT/DELETE/offline/restart/status/heal acceptance;
3. push the combined server commit to PR fix: complete per-bucket CORS release hardening #80 and require a fresh green CI set;
4. submit, build, review, merge, deploy, and publicly verify the EN/ZH documentation;
5. merge server changes, verify main CI, update this checklist, and only then close Harden per-bucket CORS replication, recovery, and compatibility after #71 #75.

Issue #76 and #77 remain the explicit boundaries for inherited Object Lock initial-sync and cross-metadata stale whole-record behavior. No release tag or image publication is authorized by this progress update.