Skip to content

perf: push cost filtering into SQL and gate vacuum - #11

Open
srinivasdrona wants to merge 1 commit into
superbasedapp:mainfrom
srinivasdrona:sql-maintenance-optimization-v2
Open

perf: push cost filtering into SQL and gate vacuum#11
srinivasdrona wants to merge 1 commit into
superbasedapp:mainfrom
srinivasdrona:sql-maintenance-optimization-v2

Conversation

@srinivasdrona

@srinivasdrona srinivasdrona commented Sep 1, 2026

Copy link
Copy Markdown

Summary

  • push cost-summary noise filtering into SQLite source queries so historical synthetic-model and all-zero token rows are discarded before Go allocates raw rows or runs per-turn rollups
  • keep the existing Go-side noise filter as a defensive backstop
  • add a guarded vacuum path for CLI/dashboard maintenance: skip full-file VACUUM below max(64 MiB, 2% of DB size) reclaimable, bound rebuilds with a 10-minute context timeout, and add observer db vacuum --force
  • document the vacuum behavior in README and CHANGELOG

Why the vacuum gate is size-aware

An absolute 64 MiB threshold is too small for very large databases: on a 40 GiB DB it would allow a full rebuild to reclaim only ~0.16% of the file. The default gate now requires the larger of 64 MiB or 2% of the DB size. For a 40 GiB DB, that means observer db vacuum skips unless roughly 819 MiB is reclaimable; --force still runs the compaction explicitly.

Performance comparison

Benchmarked against true superbasedapp/observer:main (8850462) on this machine.

SQL pushdown: BenchmarkSummaryDropsNoiseRows

Synthetic corpus: 100,000 historical noise rows + 1,000 real token rows.

Version ns/op B/op allocs/op
before ~466,341,367 ~55,622,349 ~1,924,062
after ~66,368,755 ~2,105,867 ~24,068

Approximate improvement: ~7.0x faster, ~86% lower wall time, ~96% lower bytes allocated, ~99% fewer allocations.

Maintenance gate: BenchmarkVacuumLowReclaimableDB

Low-reclaim database where the new CLI/dashboard gate skips the rebuild.

Version ns/op B/op allocs/op
before ~31,212,653 ~47,833 ~1,234
after ~21,595 ~1,776 ~49

Approximate improvement for low-value vacuum requests: ~1,445x faster, ~99.93% lower wall time, ~96% lower bytes allocated.

Validation

  • go test ./internal/intelligence/cost
  • go test ./internal/db -run 'Test(StorageStats|VacuumAndBackupInto|VacuumWithOptionsSkipsBelowReclaimableThreshold|VacuumRequiredBytesUsesLargerOfAbsoluteAndRatioThresholds)'

Note: the broader upstream internal/db package has pre-existing Windows-specific failures in TestLiveDBGate_RootResolutionNeverFailsOpen; the broader cmd/observer package does not compile in this snapshot because generated obs* symbols are absent. The touched packages/tests above pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0169310b-2e4b-4157-8609-490962d6de95
@srinivasdrona
srinivasdrona force-pushed the sql-maintenance-optimization-v2 branch from 52b2a5a to bc8559f Compare September 1, 2026 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants