Add compound, sparse, and partial index support - #173
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds full ascending compound, sparse, and partial index definitions alongside the existing single-field index support. It also folds in Mike's TM-043 SQLite modifier-update fix so the expanded unique-index support does not impose a collection-wide scan when an update leaves unique entries unchanged.
list_indexes(),index_information(), and restarts._idor declared-index candidate path even when a unique index exists.This advances the advanced-index work under #55 and addresses TM-043 from #136.
Supported partial-filter subset
Literal equality,
$eq,$exists: true,$gt,$gte,$in,$lt,$lte,$type,$and, and$or.Sparse and partial options remain mutually exclusive. Descending and hashed declarations retain their documented ascending degradation; text indexes remain skipped and TTL expiration remains unsupported.
TM-043 correctness boundary
The fast path compares semantic token sets rather than guessing from update paths. This covers compound and dotted keys, sparse or partial membership, multikey arrays,
$rename, and no-op update operators. Reordering the same multikey entries remains targeted because order does not alter uniqueness.A real unique-entry or membership change still takes the conservative full post-image validation path. That preserves within-batch conflict detection, multikey overlap checks, and atomic rollback; this PR does not claim that changed unique keys are constant-time.
Performance
Fresh baseline/current index-feature runs used 10,000 documents on the same host:
The complete bulk benchmark, including setup around the measured call, varied from 72,402 to 71,070 docs/s (-1.8%). Taken with the timed batch and repeated-run read results, this is normal run variance rather than a measurable regression.
A separate 40,000-document TM-043 smoke benchmark measured median unrelated-field point updates at about 4.22 ms without a secondary index, 4.48 ms with a non-unique index, and 4.68 ms with a unique index. Mike measured the former unique-index path at roughly 208 ms, so the collection-size-dependent penalty is absent while the unique result remains near the normal targeted path.
Validation
3839 passed, 1 skipped, 531 deselectedruff check .black --check .mypy --ignore-missing-imports tinymongogit diff --checkThe index suite covers metadata persistence and defensive copies, unique insert/update/upsert atomicity, missing-versus-null behavior, partial membership transitions, supported and rejected predicates, embedded multikey behavior, SQLite physical indexes and candidate narrowing, remote nullable membership tokens, MySQL compound materialization, and legacy catalog promotion.
The TM-043 suite additionally proves collection-size-bounded unrelated
_idupdates, zero-decode misses, same-token and reordered-multikey fast paths, compound conflicts, sparse and partial membership transitions, indexedupdate_many()candidate bounds, and atomic rollback when a unique entry really changes.