Description
Live file catalog during borg create (--list → agent → catalog-write → ClickHouse) is on the backup hot path and becomes the
bottleneck on very large trees (tens of millions of included files): agent backpressure (pipe_write), extra per-file work, multi-day
creates. #380/#394 address deadlocks/exclude flood, not this throughput case.
Proposal: optional plan (or “expensive backup”) setting to turn off live catalogization for create:
- Agent runs borg create without --list (keep --progress / --json for coarse byte progress — no per-file progress in the UI for
that plan).
- After a successful archive, BBS queues a server-side job that builds/updates the ClickHouse file catalog from the repo (same idea
as Catalog Rebuild, but automatic post-backup).
- File browser / search appear when that job finishes; create is no longer blocked on live ingest.
If “auto on expensive ops” is hard to detect reliably, a simple plan toggle (live / deferred) is enough; deferred = skip live catalog
- post-create ClickHouse sync from the archive.
Acceptance: with the option on, create wall time is close to bare borg create; catalog still becomes available via the follow-up
server job; backup succeeds even if that job is queued/retries separately.
Related: #380, #394, Discussion #7; existing Catalog Rebuild / Catalog Sync actions.
Use Case
Backup plans that include huge media/object directories (multi‑10M small files). We need create to finish in a practical time and can
accept no live per-file progress for those plans. After the archive is on the BBS server, a server-side ClickHouse catalog job is fine
— UI browse/restore can wait until that job completes. A plan-level switch (or equivalent for “expensive” backups) to disable live
catalog and defer indexing to the server would make these workloads viable without per-host agent patches.
Alternatives Considered
No response
Description
Live file catalog during borg create (--list → agent → catalog-write → ClickHouse) is on the backup hot path and becomes the
bottleneck on very large trees (tens of millions of included files): agent backpressure (pipe_write), extra per-file work, multi-day
creates. #380/#394 address deadlocks/exclude flood, not this throughput case.
Proposal: optional plan (or “expensive backup”) setting to turn off live catalogization for create:
that plan).
as Catalog Rebuild, but automatic post-backup).
If “auto on expensive ops” is hard to detect reliably, a simple plan toggle (live / deferred) is enough; deferred = skip live catalog
Acceptance: with the option on, create wall time is close to bare borg create; catalog still becomes available via the follow-up
server job; backup succeeds even if that job is queued/retries separately.
Related: #380, #394, Discussion #7; existing Catalog Rebuild / Catalog Sync actions.
Use Case
Backup plans that include huge media/object directories (multi‑10M small files). We need create to finish in a practical time and can
accept no live per-file progress for those plans. After the archive is on the BBS server, a server-side ClickHouse catalog job is fine
— UI browse/restore can wait until that job completes. A plan-level switch (or equivalent for “expensive” backups) to disable live
catalog and defer indexing to the server would make these workloads viable without per-host agent patches.
Alternatives Considered
No response