Skip to content

Add experimental sharded SQLite backend - #174

Merged
schapman1974 merged 3 commits into
masterfrom
agent/experimental-sharded-sqlite
Aug 7, 2026
Merged

Add experimental sharded SQLite backend#174
schapman1974 merged 3 commits into
masterfrom
agent/experimental-sharded-sqlite

Conversation

@schapman1974

Copy link
Copy Markdown
Owner

What changed

  • Add an opt-in sqlite-sharded backend that deterministically routes BSON _id values across independent SQLite WAL files.
  • Route exact-ID reads and writes to one shard, preserve global collection/index semantics, and enforce cross-shard uniqueness.
  • Accelerate unfiltered scans with read-only SQLite ATTACH plus ordered UNION ALL while retaining the established filtered scatter path.
  • Add synchronous, asynchronous, CLI, compatibility-report, installed-package, and cross-platform CI integration.
  • Add extensive concurrency, lifecycle, corruption, atomicity, point-read, and coverage regression tests.
  • Rework the storage benchmark so every phase uses the selected process count, supports focused backend runs, and documents baseline versus contention results in one table.

Why

Standard SQLite has one writer per database file. The experimental layout permits independent writers on different shard files without introducing a daemon, broker, or custom SQLite build. The attached-read path also removes Python-side shard re-querying and duplicate payload decoding for unfiltered scans.

Impact

The existing sqlite backend and its on-disk format are unchanged. Applications must explicitly select backend="sqlite-sharded"; the new backend remains documented as experimental, with fixed shard count and cross-file atomicity limitations.

Validation

  • .venv/bin/python -m pytest -q — passed
  • git diff --check — passed
  • Multiprocess benchmark runs validated distinct PIDs, persistence, CRUD counts, and focused backend execution.

@schapman1974
schapman1974 marked this pull request as ready for review August 7, 2026 18:57
@schapman1974
schapman1974 merged commit 6eedcea into master Aug 7, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant