Skip to content

Add compound, sparse, and partial index support - #173

Merged
schapman1974 merged 2 commits into
masterfrom
agent/compound-sparse-partial-indexes
Aug 6, 2026
Merged

Add compound, sparse, and partial index support#173
schapman1974 merged 2 commits into
masterfrom
agent/compound-sparse-partial-indexes

Conversation

@schapman1974

@schapman1974 schapman1974 commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Summary

This adds full ascending compound, sparse, and partial index definitions alongside the existing single-field index support. It also folds in Mike's TM-043 SQLite modifier-update fix so the expanded unique-index support does not impose a collection-wide scan when an update leaves unique entries unchanged.

  • Persist ordered compound keys, sparse membership, and partial filters with versioned metadata across TinyDB, SQLite, DuckDB/Parquet, PostgreSQL, and MySQL/MariaDB.
  • Enforce compound, sparse, and partial uniqueness across inserts, updates, replacements, upserts, and batch planning.
  • Preserve MongoDB membership rules: sparse indexes skip documents where every indexed field is missing but include explicit nulls; partial indexes include only matching documents.
  • Support one flat multikey field in embedded compound indexes and reject parallel arrays atomically. Remote SQL continues to reject multikey unique values when its native token constraint cannot guarantee cross-process integrity.
  • Materialize ordered compound components in SQLite, PostgreSQL, DuckDB, and MySQL native indexes while retaining exact BSON post-filtering.
  • Upgrade legacy v1 compound/sparse declarations from their former leading-field fallback when the original declaration is retried.
  • Preserve advanced index definitions through CLI collection replacement, list_indexes(), index_information(), and restarts.
  • Route SQLite modifier updates through the normal _id or declared-index candidate path even when a unique index exists.
  • Compare exact before/after unique token sets for each selected document. If all entries are unchanged, skip the full post-image scan; if any entry changes, validate the complete post-image inside the same locked transaction.
  • Update the README, changelog, benchmark notes, Talk Python contract, and roadmap.

This advances the advanced-index work under #55 and addresses TM-043 from #136.

Supported partial-filter subset

Literal equality, $eq, $exists: true, $gt, $gte, $in, $lt, $lte, $type, $and, and $or.

Sparse and partial options remain mutually exclusive. Descending and hashed declarations retain their documented ascending degradation; text indexes remain skipped and TTL expiration remains unsupported.

TM-043 correctness boundary

The fast path compares semantic token sets rather than guessing from update paths. This covers compound and dotted keys, sparse or partial membership, multikey arrays, $rename, and no-op update operators. Reordering the same multikey entries remains targeted because order does not alter uniqueness.

A real unique-entry or membership change still takes the conservative full post-image validation path. That preserves within-batch conflict detection, multikey overlap checks, and atomic rollback; this PR does not claim that changed unique keys are constant-time.

Performance

Fresh baseline/current index-feature runs used 10,000 documents on the same host:

Workload Baseline This branch Difference
One 10,000-document insert call (timed batch) 76,614 docs/s 77,222 docs/s +0.8%
Repeated 200-document insert batches 19,995 docs/s 20,639 docs/s +3.2%
Simple indexed point reads 2,223.0 q/s 2,226.6 q/s +0.2%
Complex indexed reads 76.9 q/s 77.1 q/s +0.3%

The complete bulk benchmark, including setup around the measured call, varied from 72,402 to 71,070 docs/s (-1.8%). Taken with the timed batch and repeated-run read results, this is normal run variance rather than a measurable regression.

A separate 40,000-document TM-043 smoke benchmark measured median unrelated-field point updates at about 4.22 ms without a secondary index, 4.48 ms with a non-unique index, and 4.68 ms with a unique index. Mike measured the former unique-index path at roughly 208 ms, so the collection-size-dependent penalty is absent while the unique result remains near the normal targeted path.

Validation

  • 3839 passed, 1 skipped, 531 deselected
  • 100% statement and branch coverage: 8,705 statements and 3,740 branches
  • ruff check .
  • black --check .
  • mypy --ignore-missing-imports tinymongo
  • git diff --check

The index suite covers metadata persistence and defensive copies, unique insert/update/upsert atomicity, missing-versus-null behavior, partial membership transitions, supported and rejected predicates, embedded multikey behavior, SQLite physical indexes and candidate narrowing, remote nullable membership tokens, MySQL compound materialization, and legacy catalog promotion.

The TM-043 suite additionally proves collection-size-bounded unrelated _id updates, zero-decode misses, same-token and reordered-multikey fast paths, compound conflicts, sparse and partial membership transitions, indexed update_many() candidate bounds, and atomic rollback when a unique entry really changes.

@schapman1974
schapman1974 marked this pull request as ready for review August 6, 2026 18:14
@schapman1974
schapman1974 merged commit d913eb4 into master Aug 6, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant