Audit site-replication source timestamps, tombstones, and per-site counters

[Vonng]

Context

The #75 CORS repair intentionally preserves source timestamps and deletion tombstones only for per-bucket CORS. Adversarial review confirmed that neighboring site-replication handlers and status counters use similar inherited patterns, but their metadata semantics are not identical and should not be changed without reproductions.

Audit questions

For policy, tags, SSE, quota, versioning, Object Lock, and other bucket metadata types:

• Does peer apply persist the source UpdatedAt, or replace it with local arrival time through BucketMetadataSys.Update/Delete?
• After DELETE, can the relevant getter still expose the tombstone timestamp, or can an older PUT resurrect the deleted state?
• Does SiteReplicationMetaInfo export timestamps when the live payload is nil?
• Can heal choose a default CreatedAt state as latest before applying its creation-time guard?
• Do per-site totals use the current site's field, or a cumulative count from earlier sites in the loop?
• Are delayed, reordered, duplicate, and missed events observable and diagnosable?

Known evidence to include

• Several existing totals near TotalCorsConfigCount use cumulative counters while iterating sites; an asymmetric configuration can be attributed to the wrong site.
• Several peer handlers call ordinary Update/Delete, which generate local time rather than preserving the event time.
• If a peer's bucket creation time is later than an incoming metadata event, the event is ignored but the RPC can still report success; status may retain a mismatch without an explanatory log.

Deliverables

• Add failing tests before changing each metadata type.
• Record a behavior matrix by metadata type; do not assume identical delete semantics.
• Open focused repair issues/PRs for reproduced defects.
• Introduce a shared helper only after at least two types demonstrably require the same state transition and error contract.
• Keep release, documentation, and compatibility effects explicit for every resulting repair.

This is an evidence-gathering task, not authorization for a broad replication rewrite.

[Vonng]

Reproduced behavior matrix (pending Fable adversarial review)

A review-only ObjectLayer test moved bucket creation one hour into the past, sent a peer PUT at source time T, and then sent a newer source DELETE at T+1 minute. Results reproduced on both single-drive and Erasure backends:

Type	PUT preserved T?	Newer source DELETE applied?	Current delete semantics
Policy	no; arrival time stored	no; live policy remained	supported
Tags	no; arrival time stored	no; live tags remained	supported
SSE	no; arrival time stored	no; live encryption config remained	supported
Quota	no; arrival time stored	no; live quota remained	supported
Versioning	no; arrival time stored	n/a	nil is no-op
Object Lock	no; arrival time stored	n/a	nil is no-op

Additional confirmed code findings:

• Policy exports its timestamp even when payload is nil; Tags/SSE/Quota hide deletion timestamps when payload is nil.
• Every local heal branch above writes through ordinary Update/Delete and replaces the selected source time with local arrival time.
• Tag remote heal omits UpdatedAt completely.
• Object Lock remote heal uses the wrong payload field; tracked separately in Fix Object Lock metadata field in site-replication initial sync #76.
• Heal functions seed from the first map entry before skipping creation-time defaults.
• Per-site totals for Tags/Object Lock/SSE/Versioning use cumulative counters, so results depend on site order.
• Policy total checks len(policies) > 0 on a fixed-size slice and is effectively true for every bucket.
• TotalQuotaConfigCount and ReplicatedQuotaConfig are defined by madmin but never populated.

Provisional workstream split

1. Source timestamps and tombstones: Policy/Tags/SSE/Quota delete-capable state; Versioning/Object Lock update-only state; local and remote heal through the same source-time-aware handler paths.
2. Heal selection and diagnostics: skip creation defaults before seeding; define behavior when a peer bucket was created after an incoming event; make persistent mismatch diagnosable without creating retry storms.
3. Status accounting: use each current site's payload, populate quota totals/replicated count, and avoid changing mismatch semantics.

Six types now prove a shared source-time requirement, so a small unexported site-replication apply primitive is worth evaluating. It must keep dedicated exported handlers and their type-specific nil/encoding rules, exclude lifecycle/replication special cases, reject missing metadata, default timestamps, compare strict-After, save exact source time, and avoid refactoring BucketMetadataSys.updateAndParse globally. The alternative is six direct rewrites; the requested adversarial rounds must decide which has lower long-term risk.

No production code has been changed. Final plan remains pending the requested multi-round Fable 5 Max challenge; the local Fable quota is currently exhausted.

[Vonng]

Final audit result and repair steering after four Opus 5 Max adversarial rounds

Verdict: PLAN READY.

The review-only diagnostic tests are reproductions, not production tests, and must be replaced rather than committed verbatim.

Confirmed defects

1. Arrival-time stamping: Policy, Tags, SSE, Quota, Versioning, and Object Lock peer PUTs replace source UpdatedAt with UTCNow. Policy/Tags/SSE/Quota then reject a genuinely newer source DELETE and retain the live config. Reproduced on both ObjectLayer backends.
2. Tombstone invisibility: Tags/SSE/Quota export timestamps only with live payload; Policy is already unconditional.
3. Heal selection: affected heal functions seed from a random map entry before skipping creation-default/zero states.
4. Heal apply: local heal replaces source time; Tag remote heal omits UpdatedAt; Object Lock remote field is Fix Object Lock metadata field in site-replication initial sync #76.
5. Reporting: cumulative per-site totals, policy tautologies, missing quota total/replicated fields, and malformed-config continue dropping the site's whole bucket accounting.

Lifecycle expiry remains excluded because it carries ExpiryUpdatedAt inside its merged payload. CORS remains owned by #75. Versioning/Object Lock nil stays no-op.

Required workstream order

#77-A — latest-state selection and safe heal inputs

1. Add narrow latestBucketMetaInfo(bs, timestampAccessor):
   • skip zero timestamps;
   • skip CreatedAt == config timestamp defaults before seeding;
   • select strict latest among survivors;
   • break timestamp ties by lowest deployment ID for deterministic behavior;
   • return found=false when every candidate is default/unknown.
2. Adopt it in Policy/Tags/SSE/Quota/Versioning/Object Lock heal. Do not touch CORS Harden per-bucket CORS replication, recovery, and compatibility after #71 #75.
3. Update-only Versioning/Object Lock return without mutation when selected payload is nil.
4. Add UpdatedAt: lastUpdate to Tag remote heal.
5. Quota local tombstone must use the dedicated delete path; do not Update(nil) because quota parsing leaves cached enforcement populated. Versioning must never write nil.

Commits: helper/tests, then call-site conversions. This workstream must land before any newly visible tombstone timestamps.

#77-B — source-time-aware peer apply and tombstones

1. Extend internal updateAndParse with source time and peer-require-existing inputs while preserving its existing setter switch.
2. Existing local Update/Delete pass zero/false and retain current UTCNow behavior.
3. For peer source time:
   • load/migrate/default through the existing path;
   • on ambiguous Created == zero, call GetBucketInfo only on that rare path;
   • reject missing/soft-deleted buckets without saving ghost metadata;
   • for a real legacy empty bucket, call meta.SetCreatedAt(bucketInfo.Created), then defaultTimestamps;
   • compare the field's raw current timestamp inside the same read/modify/save; skip only strict-older events;
   • persist the exact source time.
4. Add unexported updateFromPeer (parse=true) and deleteFromPeer (parse=false) returning (applied bool, err error). Exported handlers keep their error-only contract and do not log stale skips yet.
5. Policy/Tags/SSE/Quota nil uses deleteFromPeer. Versioning/Object Lock nil remains applied=false no-op.
6. Remove handler getter pre-checks, closing the current getter→Update TOCTOU and tombstone-hidden resurrection path.
7. Local heal calls dedicated handlers with the selected source time.

Then, in a separate commit and deployment gate, export deletion timestamps unconditionally for Policy/Tags/SSE/Quota.

Mixed-version hard gate: release sequencing alone is insufficient. Unconditional nil-timestamp export must be enabled only when every peer advertises #77-A-capable selection. Fall back to current payload-conditional export if any peer lacks the capability. If no capability channel can be added safely, keep the feature disabled behind explicit operator opt-in rather than defaulting it on. Document all-sites minimum versions as defence in depth.

Tests, on both ObjectLayer backends and per type:

• PUT at T persists T;
• DELETE at T+1 removes state and persists T+1;
• older PUT after tombstone is skipped with applied=false;
• equal delivery is idempotent;
• Versioning/Object Lock nil is no-op;
• absent bucket creates no ghost metadata;
• real zero-Created legacy bucket is backfilled and updated;
• exported tombstones are capability-gated in mixed-version matrices.

#77-C — status accounting

• Derive each site's totals from that site's own payload, not running counters.
• Replace policy len(policies) tautologies with actual per-site/site-count predicates.
• Populate TotalQuotaConfigCount and ReplicatedQuotaConfig (present in current madmin; server-only change).
• Ensure one malformed config does not skip TotalBucketsCount or the whole StatsSummary write for that site.
• Preserve existing mismatch predicates unless separately proven wrong.

Tests must permute site order, cover no-policy/all-policy/asymmetric cases, quota rows, and malformed payload accounting.

#77-D — persistent mismatch diagnostics

Specify before implementation:

• bounded/rate-limited signal, no per-heal-round spam or retry loop;
• report found=false as indeterminate/default-or-zero state, including legacy cases;
• consume applied=false from Audit site-replication source timestamps, tombstones, and per-site counters #77-B to explain stale/CreatedAt rejection;
• keep RPC success semantics until a deliberate compatibility decision says otherwise.

Related ordering

1. Fix Object Lock metadata field in site-replication initial sync #76 wire-field fix.
2. Preserve existing Object Lock retention during site-replication bucket adoption #78 preserve existing retention during same-name bucket adoption.
3. Audit site-replication source timestamps, tombstones, and per-site counters #77-A selection.
4. Audit site-replication source timestamps, tombstones, and per-site counters #77-B source-time apply; capability-gated tombstone export last.
5. Audit site-replication source timestamps, tombstones, and per-site counters #77-C reporting can ship independently.
6. Audit site-replication source timestamps, tombstones, and per-site counters #77-D after A/B behavior is stable.

Compatibility and release notes

• Mixed versions do not fully converge until every site runs source-time-aware apply.
• Peer-applied timestamps become source times and may look earlier than today's arrival-stamped values.
• Status policy totals will decrease to accurate values; quota rows will appear.
• Legacy zero-Created heal changes from random mutation to safe indeterminate/no-op with diagnostics.
• No storage schema migration; rollback leaves persisted source timestamps valid.

The four-round review found no remaining architecture decision. Production implementation should be split along these workstreams, not delivered as one broad PR.