Skip to content

Retain durable Git packs and add background cache repacking - #19

Open
forhappy wants to merge 2 commits into
mainfrom
codex/kubernetes-pack-storage-maintenance
Open

forhappy wants to merge 2 commits into
mainfrom
codex/kubernetes-pack-storage-maintenance

Conversation

@forhappy

@forhappy forhappy commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Large repositories exceeded the fixed 120-second native Git deadline and expanded compressed blob history into SQLite/external bodies. Retain verified receive packs and indexes durably, keep canonical identities and structural objects in SQLite, and restore packs directly after restart. Add supervised background repacking of serving caches without deleting files beneath active clone/fetch readers.

  • Verify every pack member against its canonical object identity and digest, including existing objects and thin-pack bases. Incomplete packs use a private cache and bounded, verified body extraction.
  • Batch object metadata and graph certificates while preserving typed dependency checks, ref authorization, and atomic publication.
  • Run one separately admitted maintenance job per process, checking resident repositories every 60 seconds and rotating verified cache generations at 1,024 loose objects or eight packs.
  • Include durable packs/indexes in backup verification and restore. Add an isolated persistent local server helper and reproducible repository qualification script.

Validation: 114 library tests passed serially on the implementation before reconciling recent upstream changes. A measured predecessor passed a pristine 29,839-object Kubernetes tree fixture: both clone protocols, strict fsck, incremental push/fetch/pull, and fresh-workspace recovery; cold clone took 7.293 seconds. Its background repack fixture reduced cache bytes from 356,499 to 146,715 in 0.376 seconds, followed by a passing clone/fsck. Committed evidence records the exact measured binary and distinguishes it from the newer oversized-blob path.

This PR is a draft. Complete upstream Kubernetes history (162,267 commits, 1,812,898 objects, 1,312 branches/tags) qualification is in progress. The initial concurrent library run exposed a native-fence cleanup assertion that passed serially; concurrent cancellation remains a qualification item. Recent main changes are being reconciled. Durable pack GC/compaction, schema upgrades, and provider-fault/concurrent-load release gates remain open. No production capacity or latency SLA is claimed.

@forhappy
forhappy marked this pull request as ready for review October 2, 2026 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant