Skip to content

feat: add bulk upsert by external_id (#102)#109

Merged
dcfocus merged 3 commits into
lance-format:mainfrom
dcfocus:feat/issue-102-bulk-upsert
Jun 27, 2026
Merged

feat: add bulk upsert by external_id (#102)#109
dcfocus merged 3 commits into
lance-format:mainfrom
dcfocus:feat/issue-102-bulk-upsert

Conversation

@dcfocus

@dcfocus dcfocus commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a bulk / batch upsert (insert-or-replace by external_id) API across every layer, closing #102. Previously every upsert path was single-record, and add_many rejects existing external_ids, so there was no way to idempotently upsert a batch in one operation.

Stacked on #108 (issue #100). This branch builds on the indexed-validation work; review/merge #108 first. Until then this PR shows both commits — the batch-upsert commit is the last one. The diff will reduce to just #102 once #108 lands.

Core

ContextStore::upsert_many_by_external_id(records) -> Vec<UpsertResult> applies insert-or-replace per external_id in one logical operation, returning one result per input record (in order) indicating inserted vs. replaced, the resulting id, and the final version.

Semantics mirror the single-record upsert_by_external_id exactly:

  • every record must carry a non-empty external_id;
  • duplicate ids / external_ids within the batch are rejected (parity with add_many);
  • validation is all-or-nothing — nothing is written if any record is invalid;
  • caller-supplied supersession fields are ignored (replacement is managed by external_id);
  • an insert whose external_id already exists on a non-tombstone but hidden row is rejected, exactly as a single insert is.

Composes with #100: a batch resolves existing external_ids in a single scan and validates id uniqueness via the indexed path (find_existing_keys), instead of full-scanning per record. The whole batch is written in one pass (single version bump where records share a shard). Supersession is resolved within the external_id-filtered set, which is correct for every flow that creates supersession through the public API (a successor keeps its predecessor's external_id).

Parity across surfaces

Layer Addition
lance-context-api ContextStoreApi::upsert_many + UpsertRecordsRequest / UpsertRecordsResponse / UpsertResultDto
lance-context-core upsert_many_by_external_id + api_impl dispatch
lance-context (unified) dispatch to local/remote
lance-context-client RemoteContextStore::upsert_many + ContextClient::upsert_records
REST server PUT /api/v1/contexts/{name}/records/batch
Python Context.upsert_many(records, key="external_id") (+ pyo3 binding)

Tests

  • core: insert, replace, mixed, idempotent re-application, within-batch duplicate id/external_id, missing/empty external_id, id collision with store, empty batch, and the BTree-indexed path.
  • REST: insert+replace batch, empty rejected, missing external_id rejected.
  • Python: python/tests/test_upsert_many.py (insert/replace/idempotent/mixed/within-batch dup/missing external_id/empty).
  • README: bulk-upsert example added next to add_many.

Checks

  • cargo test -p lance-context-core --lib — 65 passed (1 ignored bench)
  • cargo test -p lance-context-server — 17 passed
  • cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings — clean
  • Python: new test_upsert_many.py (6) + test_add_many.py + test_external_id.py pass

Note: ~24 pre-existing Python test failures on main (mock _DummyInner.add() argument-count drift in the single add/upsert paths and S3 network tests) are unrelated to this change — this PR only adds upsert_many and does not touch those paths.

Acceptance criteria (#102)

  • Upsert a batch keyed by external_id in one call (new inserted, existing replaced/superseded), single version bump where possible
  • Per-record results indicate inserted vs. replaced and the resulting id
  • Within-batch duplicate external_id handling defined and tested (rejected)
  • Parity across Python / core / REST / client
  • Tests cover insert, replace, mixed batches, and idempotent re-application
  • Composes with Uniqueness validation full-scans the dataset on every write (O(n²) for incremental/bulk appends) #100 so batch upsert does not full-scan per record

Closes #102

dcfocus added 3 commits June 24, 2026 15:43
Uniqueness validation previously listed and deserialized the entire
dataset (all columns, all history including expired/retired/superseded/
tombstoned rows) into a Vec<ContextRecord> on every add / add_many /
upsert / update, then linear-scanned for collisions. That made each
write O(N) in store size and incremental/streaming appends O(N^2).

Replace the full scan with find_existing_keys(), which issues projected,
filtered LsmScanner scans over only the candidate keys:

- `id IN (...)` projecting just `id` + `content_type`
- `external_id IN (...)` projecting just `external_id` + `content_type`

Filters are pushed down to each data source, so the id lookup is
index-accelerated when an id scalar index is configured, and neither
scan deserializes full records or materializes the whole dataset.

Behavior is preserved exactly: within-batch duplicate detection and the
existing error messages are unchanged; tombstones remain excluded from
collisions (a key whose only surviving row is a tombstone is reusable)
while superseded / retired / expired non-tombstone rows still reserve
their key. The external_id scan is guarded by a schema check so datasets
without that column are unaffected. Caller-supplied external_ids are
escaped before going into the IN filter.

Tests: id/external_id collisions, within-batch dups, reuse-after-delete,
reject-after-supersede, the indexed-scan path, large-store correctness,
single-quote escaping, and an ignored append-cost benchmark (store grew
~20x while per-call append cost grew ~4x, not ~20x).

Closes lance-format#100
Adds an insert-or-replace-by-`external_id` batch API across every layer.
Previously every upsert path was single-record and `add_many` rejected
existing `external_id`s, so there was no way to idempotently upsert a
batch in one operation.

Core: `ContextStore::upsert_many_by_external_id(records)` applies
insert-or-replace per `external_id` in one logical operation and returns
one `UpsertResult` per input record (inserted vs. replaced, resulting
id), all carrying the final version.

Semantics (parity with the single-record `upsert_by_external_id`):
- every record must carry a non-empty `external_id`;
- duplicate `id`s / `external_id`s within the batch are rejected
  (parity with `add_many`);
- validation is all-or-nothing — nothing is written if any record is
  invalid;
- caller-supplied supersession fields are ignored (replacement is
  managed by `external_id`);
- an insert whose `external_id` already exists on a non-tombstone but
  hidden row is rejected, exactly as a single insert is.

Composes with the lance-format#100 indexed validation: a batch resolves existing
`external_id`s in a single scan and validates `id` uniqueness via the
indexed path, rather than full-scanning per record. The whole batch is
written in one pass (single version bump where records share a shard).

Plumbed through every surface for parity:
- api: `ContextStoreApi::upsert_many` + `UpsertRecordsRequest` /
  `UpsertRecordsResponse` / `UpsertResultDto`;
- core dispatch (`api_impl`), unified store, and remote client;
- REST: `PUT /contexts/{name}/records/batch`;
- Python: `Context.upsert_many(records, key="external_id")` (+ binding).

Tests: core (insert / replace / mixed / idempotent / within-batch dup /
missing external_id / id collision / empty / indexed path), REST
(insert+replace, empty, missing external_id), and Python
(`test_upsert_many.py`). README gains a bulk-upsert example.

Closes lance-format#102
@dcfocus dcfocus merged commit c83869f into lance-format:main Jun 27, 2026
9 checks passed
dcfocus added a commit that referenced this pull request Jun 27, 2026
)

## Summary

Adds a retrieval-quality evaluation harness that measures
`search`/`retrieve` quality against a labeled query set and compares
quality across dataset versions — closing #98. Cross-version regression
eval is the differentiator a stateless vector DB can't offer.

This PR is **independent** of #108 (#100) and #109 (#102) — it branches
from `main` and can be reviewed/merged on its own.

## Core (`lance-context-core::eval`)

- **Eval dataset format** — `EvalQuerySet` / `EvalQuery` /
`RelevanceLabel`, referencing records by stable `external_id` with
optional graded relevance, loadable from / serializable to **JSONL**.
- **Metrics** — recall@k, precision@k, MRR, nDCG@k (linear gain),
hit-rate, as pure functions.
- **Runner** — `ContextStore::evaluate(query_set, config)` scores a set
at the current version for **vector** (`search`) or **hybrid**
(`retrieve`) mode, with configurable `k`, filters, and lifecycle
options. Returns per-query + aggregate scores and a manifest (query-set
id, version, k, mode, distance metric).
- **Version A/B** — `ContextStore::evaluate_versions(..., baseline,
candidate)` checks out each version, evaluates, restores the original
version, and reports per-metric deltas. `MetricScores::delta` also
enables config A/B (two `evaluate` runs).

## Python

- `Context.evaluate(queries, *, k, mode, filters, include_expired,
include_retired)` → report dict.
- `Context.evaluate_versions(queries, baseline_version,
candidate_version, ...)` → `{baseline, candidate, deltas}`.

Query sets and reports are marshaled as JSON across the pyo3 boundary,
reusing the serde-derived core types.

## Tests

- **core** (12): metric correctness vs hand-computed fixtures (perfect /
rank-2 / none / graded nDCG / precision@k), JSONL round-trip, vector +
hybrid runners, **filter** and **lifecycle** handling, and the
version-A/B mechanism (zero-delta + version restore).
- **python** (5): `test_eval.py` — vector mode, graded nDCG, config A/B
delta, version A/B, invalid-mode error.

README gains an eval + version-A/B example.

## Checks

- `cargo test -p lance-context-core --lib` — 59 passed
- `cargo fmt --all -- --check` and `cargo clippy --workspace
--all-targets -- -D warnings` — clean
- `ruff format --check`, `ruff check`, `pyright` — clean
- `test_eval.py` — 5 passed

## Acceptance criteria (#98)

- [x] Run a labeled query set → recall@k / MRR / nDCG for vector and
hybrid retrieval
- [x] A/B two versions (or two configs) → per-metric deltas
- [x] Results pinned to a context version + config in a reproducible
report
- [x] Tests cover metric correctness against fixtures, filter/lifecycle
handling, and the version-A/B mechanism

## Notes

- Cross-version A/B with *differing data* is not unit-tested because
MemWAL-backed writes don't reliably advance the base-table manifest
version (the same limitation `test_time_travel_checkout` `xfail`s on).
The A/B **mechanism** (checkout both versions, restore, per-metric
deltas) and metric-difference detection are covered via the same-version
and config-A/B tests.
- The eval-set format is intentionally simple JSONL so it can be shared
with the post-training pipeline's eval/holdout set (#96 / #103), per the
issue notes.
- nDCG uses linear gain (`grade / log2(rank+1)`); documented on
`MetricScores`.

Closes #98
@dcfocus dcfocus deleted the feat/issue-102-bulk-upsert branch June 27, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add bulk / batch upsert (insert-or-replace by external_id)

1 participant