Skip to content

Accelerate SQLite bulk inserts - #170

Merged
schapman1974 merged 1 commit into
masterfrom
agent/sqlite-insert-fast-path
Aug 5, 2026
Merged

Accelerate SQLite bulk inserts#170
schapman1974 merged 1 commit into
masterfrom
agent/sqlite-insert-fast-path

Conversation

@schapman1974

Copy link
Copy Markdown
Owner

Summary

This PR speeds up SQLite-backed bulk inserts without changing the MongoDB-compatible behavior exposed by TinyMongo.

  • Adds exact-built-in BSON encoding and identity-key fast paths while preserving subclass handling for types such as Code and Binary.
  • Prepares each SQLite insert once, reusing its serialized payload and typed physical _id across candidate checks, retries, and commits.
  • Adds an optimistic initial-load path for empty, canonical SQLite tables.
  • Avoids repeatedly probing collections already known to be nonempty.
  • Expands coverage around ordered/unordered failures, retries, timestamps, typed IDs, custom mappings, schema changes, triggers, and unique indexes.

Why

Profiling showed that most bulk-insert time was spent above SQLite: repeated BSON checks, JSON serialization, identity-key construction, and candidate planning. The new prepared-row path removes that duplicate work. For a brand-new compatible table, the optimistic path also skips the general conflict planner after proving under a write transaction that the table is empty and safe.

Contract and safety boundaries

The optimistic path is deliberately narrow:

  • It accepts only exact built-in document shapes that have already passed BSON validation.
  • It is disabled for top-level zero timestamps that require server-time stamping.
  • It begins an immediate transaction and rechecks the index catalog, schema, and emptiness inside that transaction.
  • It is disabled for unique catalog specs, triggers, external unique indexes, legacy/custom table definitions, and noncanonical primary-key behavior.
  • Any SQLite integrity failure, including commit-time failure, rolls back and falls back to the existing Mongo-compatible planner.
  • Nonempty collections retain the prepared general path, including ordered and unordered error semantics.

Local benchmark

Median local throughput for 10,000-document batches:

Engine/path Inserts/sec
TinyMongo SQLite on master 13,123
TinyMongo SQLite, nonempty prepared path 63,486
TinyMongo SQLite, initial empty-table path 70,250
MongoDB (w=1, j=true) 108,890
Direct SQLite 258,821

The initial-load path is about 5.35x faster than the previous TinyMongo SQLite baseline and about 19% faster than the prepared-only stage. MongoDB results were noisier across runs; its comparison explicitly used acknowledged, journaled writes. Direct SQLite is an implementation ceiling rather than a contract-equivalent comparison.

Validation

  • Full suite: 3744 passed, 1 skipped, 531 deselected
  • Coverage: 100% across 8,314 statements and 3,524 branches
  • ruff check .
  • black --check .
  • git diff --check

@schapman1974
schapman1974 marked this pull request as ready for review August 5, 2026 18:43
@schapman1974
schapman1974 merged commit f70b24a into master Aug 5, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant