Sync engine metadata grows to ~10× the user database

[KayLeung]

The sidecar sync meta DB is 757MB compared to 70MB for the production DB.

I had AI research this, and it suggests custom key-prefix encoding + zlib compression as a fix (e.g. shortening/removing repeated prefixes like sqlitedata_icloud_userModificationTime_).

some quick benchmark( ~70MB DB )

Full DB:

Partial:

For reference, SwiftData compresses its store CKRecord with LZFSE.

I'd like to find a fix upstream, or a hook point that lets us apply this without forking the project.

Thanks.

[mbrandonw]

Hi @KayLeung, thanks for the report. I'm not sure how to square this though:

Compressing just lastKnownServerRecord saves only 4mb, which makes sense since that is only the system fields, and compressing just _lastKnownServerRecordAllFields saves 350mb, which also makes sense since that is the full record. Yet somehow compressing both leads to a savings of 573mb, which is 219mb more than each savings combined.

Further, in the first image, which compression image is being used?