feat: add bulk upsert by external_id (#102)#109
Merged
Conversation
Uniqueness validation previously listed and deserialized the entire dataset (all columns, all history including expired/retired/superseded/ tombstoned rows) into a Vec<ContextRecord> on every add / add_many / upsert / update, then linear-scanned for collisions. That made each write O(N) in store size and incremental/streaming appends O(N^2). Replace the full scan with find_existing_keys(), which issues projected, filtered LsmScanner scans over only the candidate keys: - `id IN (...)` projecting just `id` + `content_type` - `external_id IN (...)` projecting just `external_id` + `content_type` Filters are pushed down to each data source, so the id lookup is index-accelerated when an id scalar index is configured, and neither scan deserializes full records or materializes the whole dataset. Behavior is preserved exactly: within-batch duplicate detection and the existing error messages are unchanged; tombstones remain excluded from collisions (a key whose only surviving row is a tombstone is reusable) while superseded / retired / expired non-tombstone rows still reserve their key. The external_id scan is guarded by a schema check so datasets without that column are unaffected. Caller-supplied external_ids are escaped before going into the IN filter. Tests: id/external_id collisions, within-batch dups, reuse-after-delete, reject-after-supersede, the indexed-scan path, large-store correctness, single-quote escaping, and an ignored append-cost benchmark (store grew ~20x while per-call append cost grew ~4x, not ~20x). Closes lance-format#100
Adds an insert-or-replace-by-`external_id` batch API across every layer. Previously every upsert path was single-record and `add_many` rejected existing `external_id`s, so there was no way to idempotently upsert a batch in one operation. Core: `ContextStore::upsert_many_by_external_id(records)` applies insert-or-replace per `external_id` in one logical operation and returns one `UpsertResult` per input record (inserted vs. replaced, resulting id), all carrying the final version. Semantics (parity with the single-record `upsert_by_external_id`): - every record must carry a non-empty `external_id`; - duplicate `id`s / `external_id`s within the batch are rejected (parity with `add_many`); - validation is all-or-nothing — nothing is written if any record is invalid; - caller-supplied supersession fields are ignored (replacement is managed by `external_id`); - an insert whose `external_id` already exists on a non-tombstone but hidden row is rejected, exactly as a single insert is. Composes with the lance-format#100 indexed validation: a batch resolves existing `external_id`s in a single scan and validates `id` uniqueness via the indexed path, rather than full-scanning per record. The whole batch is written in one pass (single version bump where records share a shard). Plumbed through every surface for parity: - api: `ContextStoreApi::upsert_many` + `UpsertRecordsRequest` / `UpsertRecordsResponse` / `UpsertResultDto`; - core dispatch (`api_impl`), unified store, and remote client; - REST: `PUT /contexts/{name}/records/batch`; - Python: `Context.upsert_many(records, key="external_id")` (+ binding). Tests: core (insert / replace / mixed / idempotent / within-batch dup / missing external_id / id collision / empty / indexed path), REST (insert+replace, empty, missing external_id), and Python (`test_upsert_many.py`). README gains a bulk-upsert example. Closes lance-format#102
4 tasks
dcfocus
added a commit
that referenced
this pull request
Jun 27, 2026
) ## Summary Adds a retrieval-quality evaluation harness that measures `search`/`retrieve` quality against a labeled query set and compares quality across dataset versions — closing #98. Cross-version regression eval is the differentiator a stateless vector DB can't offer. This PR is **independent** of #108 (#100) and #109 (#102) — it branches from `main` and can be reviewed/merged on its own. ## Core (`lance-context-core::eval`) - **Eval dataset format** — `EvalQuerySet` / `EvalQuery` / `RelevanceLabel`, referencing records by stable `external_id` with optional graded relevance, loadable from / serializable to **JSONL**. - **Metrics** — recall@k, precision@k, MRR, nDCG@k (linear gain), hit-rate, as pure functions. - **Runner** — `ContextStore::evaluate(query_set, config)` scores a set at the current version for **vector** (`search`) or **hybrid** (`retrieve`) mode, with configurable `k`, filters, and lifecycle options. Returns per-query + aggregate scores and a manifest (query-set id, version, k, mode, distance metric). - **Version A/B** — `ContextStore::evaluate_versions(..., baseline, candidate)` checks out each version, evaluates, restores the original version, and reports per-metric deltas. `MetricScores::delta` also enables config A/B (two `evaluate` runs). ## Python - `Context.evaluate(queries, *, k, mode, filters, include_expired, include_retired)` → report dict. - `Context.evaluate_versions(queries, baseline_version, candidate_version, ...)` → `{baseline, candidate, deltas}`. Query sets and reports are marshaled as JSON across the pyo3 boundary, reusing the serde-derived core types. ## Tests - **core** (12): metric correctness vs hand-computed fixtures (perfect / rank-2 / none / graded nDCG / precision@k), JSONL round-trip, vector + hybrid runners, **filter** and **lifecycle** handling, and the version-A/B mechanism (zero-delta + version restore). - **python** (5): `test_eval.py` — vector mode, graded nDCG, config A/B delta, version A/B, invalid-mode error. README gains an eval + version-A/B example. ## Checks - `cargo test -p lance-context-core --lib` — 59 passed - `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets -- -D warnings` — clean - `ruff format --check`, `ruff check`, `pyright` — clean - `test_eval.py` — 5 passed ## Acceptance criteria (#98) - [x] Run a labeled query set → recall@k / MRR / nDCG for vector and hybrid retrieval - [x] A/B two versions (or two configs) → per-metric deltas - [x] Results pinned to a context version + config in a reproducible report - [x] Tests cover metric correctness against fixtures, filter/lifecycle handling, and the version-A/B mechanism ## Notes - Cross-version A/B with *differing data* is not unit-tested because MemWAL-backed writes don't reliably advance the base-table manifest version (the same limitation `test_time_travel_checkout` `xfail`s on). The A/B **mechanism** (checkout both versions, restore, per-metric deltas) and metric-difference detection are covered via the same-version and config-A/B tests. - The eval-set format is intentionally simple JSONL so it can be shared with the post-training pipeline's eval/holdout set (#96 / #103), per the issue notes. - nDCG uses linear gain (`grade / log2(rank+1)`); documented on `MetricScores`. Closes #98
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a bulk / batch upsert (insert-or-replace by
external_id) API across every layer, closing #102. Previously every upsert path was single-record, andadd_manyrejects existingexternal_ids, so there was no way to idempotently upsert a batch in one operation.Core
ContextStore::upsert_many_by_external_id(records) -> Vec<UpsertResult>applies insert-or-replace perexternal_idin one logical operation, returning one result per input record (in order) indicating inserted vs. replaced, the resulting id, and the final version.Semantics mirror the single-record
upsert_by_external_idexactly:external_id;ids /external_ids within the batch are rejected (parity withadd_many);external_id);external_idalready exists on a non-tombstone but hidden row is rejected, exactly as a single insert is.Composes with #100: a batch resolves existing
external_ids in a single scan and validatesiduniqueness via the indexed path (find_existing_keys), instead of full-scanning per record. The whole batch is written in one pass (single version bump where records share a shard). Supersession is resolved within theexternal_id-filtered set, which is correct for every flow that creates supersession through the public API (a successor keeps its predecessor'sexternal_id).Parity across surfaces
lance-context-apiContextStoreApi::upsert_many+UpsertRecordsRequest/UpsertRecordsResponse/UpsertResultDtolance-context-coreupsert_many_by_external_id+api_impldispatchlance-context(unified)lance-context-clientRemoteContextStore::upsert_many+ContextClient::upsert_recordsPUT /api/v1/contexts/{name}/records/batchContext.upsert_many(records, key="external_id")(+ pyo3 binding)Tests
id/external_id, missing/emptyexternal_id,idcollision with store, empty batch, and the BTree-indexed path.external_idrejected.python/tests/test_upsert_many.py(insert/replace/idempotent/mixed/within-batch dup/missing external_id/empty).add_many.Checks
cargo test -p lance-context-core --lib— 65 passed (1 ignored bench)cargo test -p lance-context-server— 17 passedcargo fmt --all -- --checkandcargo clippy --workspace --all-targets -- -D warnings— cleantest_upsert_many.py(6) +test_add_many.py+test_external_id.pypassAcceptance criteria (#102)
external_idin one call (new inserted, existing replaced/superseded), single version bump where possibleexternal_idhandling defined and tested (rejected)Closes #102