perf: push cost filtering into SQL and gate vacuum - #11
Open
srinivasdrona wants to merge 1 commit into
Open
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0169310b-2e4b-4157-8609-490962d6de95
srinivasdrona
force-pushed
the
sql-maintenance-optimization-v2
branch
from
September 1, 2026 17:46
52b2a5a to
bc8559f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VACUUMbelowmax(64 MiB, 2% of DB size)reclaimable, bound rebuilds with a 10-minute context timeout, and addobserver db vacuum --forceWhy the vacuum gate is size-aware
An absolute 64 MiB threshold is too small for very large databases: on a 40 GiB DB it would allow a full rebuild to reclaim only ~0.16% of the file. The default gate now requires the larger of 64 MiB or 2% of the DB size. For a 40 GiB DB, that means
observer db vacuumskips unless roughly 819 MiB is reclaimable;--forcestill runs the compaction explicitly.Performance comparison
Benchmarked against true
superbasedapp/observer:main(8850462) on this machine.SQL pushdown:
BenchmarkSummaryDropsNoiseRowsSynthetic corpus: 100,000 historical noise rows + 1,000 real token rows.
Approximate improvement: ~7.0x faster, ~86% lower wall time, ~96% lower bytes allocated, ~99% fewer allocations.
Maintenance gate:
BenchmarkVacuumLowReclaimableDBLow-reclaim database where the new CLI/dashboard gate skips the rebuild.
Approximate improvement for low-value vacuum requests: ~1,445x faster, ~99.93% lower wall time, ~96% lower bytes allocated.
Validation
go test ./internal/intelligence/costgo test ./internal/db -run 'Test(StorageStats|VacuumAndBackupInto|VacuumWithOptionsSkipsBelowReclaimableThreshold|VacuumRequiredBytesUsesLargerOfAbsoluteAndRatioThresholds)'Note: the broader upstream
internal/dbpackage has pre-existing Windows-specific failures inTestLiveDBGate_RootResolutionNeverFailsOpen; the broadercmd/observerpackage does not compile in this snapshot because generatedobs*symbols are absent. The touched packages/tests above pass.