Accelerate indexed SQLite complex reads

[schapman1974]

Summary

This adds a conservative candidate-selection path for complex reads on the TinyMongo SQLite backend.

TinyMongo now reuses declared top-level SQLite expression indexes to narrow positive $and queries that have a scalar equality or $in anchor. Safe numeric range and $mod predicates are also applied in SQLite before BSON decoding. The shared Python BSON matcher remains the final authority, so candidate selection cannot change the MongoDB-facing result.

No collection-table columns or new storage format are introduced. Existing TinyMongo index metadata and expression indexes are reused, with missing scalar/type companion indexes reconciled lazily.

Why

Complex Mongo-style predicates previously fell back to decoding and testing every document in a SQLite collection, even when the application had already declared a useful index. Large payloads made that substantially slower than the underlying database needed to be.

This change treats SQLite filtering as a conservative pre-query: it reduces the rows transferred and decoded while preserving the established matcher for exact MongoDB semantics.

Implementation details

• Flattens only positive implicit/explicit $and predicates; it never narrows through $or, $nor, or negative-only expressions.
• Uses type-disjoint indexed branches for bool, number, string, and ambiguous array/object candidates.
• Preserves MongoDB scalar equality against array members by including container candidates for exact post-filtering.
• Pushes safe top-level numeric $gt/$gte/$lt/$lte and $mod predicates into SQLite.
• Keeps arrays, objects, Decimal128, nonfinite numbers, values outside the safe numeric range, unsupported operators, and dotted fields on conservative paths.
• Applies exact matching before projection, skip, limit, or counting.
• Orders candidates by physical row ID so natural-order cursor behavior is unchanged.
• Caps the complete parameter set below SQLite's portable bind-variable boundary.
• Rebuilds missing physical/type expression indexes from durable catalog metadata when needed.

Performance

Reproducible warmed comparison:

• 10,000 identical documents
• 200 complex queries after 3 warmups
• one declared group index
• identical $and(group $in, i range, i $mod) filter
• all engines fully decoded and validated the same 215 rows per query

Engine	Complex queries/sec	Average latency
TinyMongo SQLite before	~6.4	~156 ms
TinyMongo SQLite with this change	73.9	13.538 ms
MongoDB	136.0	7.354 ms
Raw SQLite	361.2	2.768 ms

The TinyMongo path is approximately 11.5x faster than the prior full-scan fallback. MongoDB used acknowledged journaled writes (w=1, j=true) during setup; writes were not part of the timed read loop.

The new benchmark is available at tests/benchmarks/bench_sqlite_complex_reads.py.

Compatibility coverage

Focused tests cover:

• mixed bool/number/string $in values
• arrays and embedded objects
• Decimal128 and huge integers
• numeric boundaries around ±2^53 and signed int64
• negative modulo behavior
• unsupported and negative-only fallback queries
• natural ordering with interleaved scalar/container candidates
• projection, count, skip, and limit
• missing catalogs and externally removed native indexes
• SQLite parameter limits and overflow fallback
• concurrent-safe index reconciliation

Validation

• 3763 passed, 1 skipped, 531 deselected
• 100% statement and branch coverage across tinymongo/*
• 314 shared SQLite/MongoDB sync and async contract checks passed, with 2 expected MongoDB skips
• ruff check .
• black --check .
• mypy --ignore-missing-imports tinymongo

The roadmap records the completed candidate-selective read work and the measured comparison.