kv-lance: native lance read/write backend (drop hand-rolled LSM)

.claude/board/AGENT_LOG.md

@@ -132,3 +132,162 @@ EpisodicWitness64; replace BindSpace; wire deprecated→cognitive-shader-driver.
   files), so hand-formatted to match surrounding `lance/` style.
 
 **Commit(s):** (this commit)
+
+## 2026-05-30 — Step-2 gridlake: orchestrated build + savant review + BLOCKER fix
+**Branch:** claude/sleepy-cori-aRK2x
+**Scope:**
+- `kvs/lance/{mod,flusher,schema}.rs`, `kvs/config.rs`, `kvs/lance/tests.rs`
+- `.claude/lance-backend/GRIDLAKE.md`, `.rustfmt.toml`, board logs
+**Verdict:** PASS
+
+**What was done (max 5 lines):**
+- Orchestrated Opus agents via file-based A2A (tee -a logs): DOC → 800-line
+  GRIDLAKE architecture; CODE → P1 adaptive batching, P2 WAL atomic-recovery
+  test, P3 per-row `seq` column.
+- 3 read-only savants (no cargo; orchestrator sole cargo runner) reviewed the
+  diff: S2/S3 clean, S1 found 1 BLOCKER + 3 MAJOR + 4 MINOR + 1 NIT.
+- Fixed BLOCKER (seq seeded from persisted Lance max), real length checks,
+  `flusher_tick_interval` knob + deterministic coalescing test, NITs; +2
+  regression tests; documented accepted limitations.
+- `.rustfmt.toml` made stable-honest (org 99%-stable policy).
+
+**Tests run (orchestrator):**
+- `cargo check --features kv-lance --tests` → Finished, 0 errors
+- `cargo test --features kv-lance --lib kvs::lance` → 98 passed, 0 failed, 3 ignored
+
+**Open questions / follow-ups:**
+- Schema migration for pre-`seq` datasets; persist seq in WAL for exact replay;
+  per-commit seq on the gate path; max-seq via manifest metadata not a scan.
+
+**Commit(s):** (this commit)
+
+## 2026-05-30T14:40 — phase3-step1 (full-auto session)
+**Branch:** claude/sleepy-cori-aRK2x
+**Scope:**
+- surrealdb/core/src/kvs/config.rs (WritePath::LsmColumnar variant)
+- surrealdb/core/src/kvs/lance/mod.rs (exhaustive write_path dispatch)
+- surrealdb/core/src/kvs/lance/tests.rs (writepath_lsm_columnar_smoke)
+**Verdict:** PASS
+
+**What was done (max 5 lines):**
+- Added the opt-in `WritePath::LsmColumnar` variant (GRIDLAKE §8 Phase 3 seam).
+- Wired it through every write_path match in mod.rs via or-patterns with
+  LsmWithWal (commit dispatch, get + scan_impl snapshot selection, the two
+  read-path `== LsmWithWal` checks) — currently aliases the proven LSM hot path.
+- The single-pass columnar flush builder lands in step-2 behind this seam.
+
+**Tests run:**
+- `cargo check -p surrealdb-core --features kv-lance --tests` → Finished, 6 pre-existing warnings
+- `cargo test -p surrealdb-core --features kv-lance --lib kvs::lance` → 99 passed; 0 failed; 3 ignored
+
+**Open questions / follow-ups:**
+- Step-2: FlusherConfig.columnar flag → do_flush branch → build_columnar_merge_batch
+  (single up-front-sized Arrow builder pass) + extracted execute_merge; parity test.
+
+**Commit(s):** 00f0e12
+
+## 2026-05-30T14:58 — phase3-step2 (full-auto session)
+**Branch:** claude/sleepy-cori-aRK2x
+**Scope:**
+- surrealdb/core/src/kvs/lance/flusher.rs (columnar flag, do_flush branch,
+  execute_merge extraction, build_columnar_merge_batch)
+- surrealdb/core/src/kvs/lance/mod.rs (spawn columnar flag)
+- surrealdb/core/src/kvs/config.rs (LsmColumnar doc)
+- surrealdb/core/src/kvs/lance/tests.rs (writepath_lsm_columnar_flush_persists)
+**Verdict:** PASS
+
+**What was done (max 5 lines):**
+- LsmColumnar flush now builds the merge source in ONE up-front-sized Arrow
+  columnar pass over the snapshot (build_columnar_merge_batch) — one fused
+  batch (live + tombstone rows), no row-vec partition, no two-batch concat.
+- FlusherConfig.columnar flag (set from write_path at spawn) branches do_flush;
+  MergeInsertBuilder execution extracted into shared execute_merge.
+- Row path unchanged + default. memtable/WAL stay row-oriented (full SoA = future).
+
+**Tests run:**
+- `cargo test -p surrealdb-core --features kv-lance --lib kvs::lance` → 100 passed; 0 failed; 3 ignored
+
+**Open questions / follow-ups:**
+- GRIDLAKE §6.2 native-Arrow memtable+WAL (SoA, CONJECTURE) still open — this
+  step is the flush-side columnar build only, not a natively columnar memtable.
+- Phase 4 (tombstone GC + version backpressure) is the next roadmap item.
+
+**Commit(s):** d9bfca7
+
+## 2026-05-30T15:30 — phase3 clippy hygiene (full-auto session)
+**Branch:** claude/sleepy-cori-aRK2x
+**Scope:** surrealdb/core/src/kvs/lance/mod.rs (get, scan_impl)
+**Verdict:** PASS
+
+**What was done (max 5 lines):**
+- Ran clippy on the kv-lance surface (9m34s); exit 0. My Phase 3 changes
+  introduced ZERO new lints — the 3 it cited are pre-existing (verified vs
+  348bb4d), just inside the read-path fns I edited for LsmColumnar.
+- Cleared them with clippy's verbatim fixes: get() nested if-let → let-chain;
+  scan_impl() two `match {Ok=>Some,Err=>None}` → `.ok()`. Behaviour-identical.
+
+**Tests run:**
+- `cargo test -p surrealdb-core --features kv-lance --lib kvs::lance` → 100 passed; 0 failed; 3 ignored
+
+**Open questions / follow-ups:**
+- 6 clippy warnings remain: unwired TimelineView dead-code (prior session,
+  intentional). `-D warnings` can't pass until that consumer lands — out of scope.
+- Integration suite (SURREAL_TEST_KV=lance) NOT run: full-workspace build risks
+  ENOSPC (13G free); needs more reclaim or a scoped run.
+
+**Commit(s):** (this commit)
+
+## 2026-05-30T16:10 — kv-lance NATIVE REWRITE: orchestration start (full-auto)
+**Branch:** claude/sleepy-cori-aRK2x
+**Scope:** .claude/lance-backend/specs/{00,10,20,30}-*.md
+**Verdict:** IN PROGRESS
+**Plan:** delete hand-rolled LSM (memtable/wal/flusher/commit_gate + WritePath);
+rewire kv-lance to native lance read/write (MergeInsert commit, checkout_version
+reads, lance optimize) — same path lance-graph uses. Pipeline: 1 Opus agent/file
+(mod.rs, config.rs, tests.rs) → savant testers → fix → strip `// ///REVIEW:` →
+clippy (sole gate) → PR → subscribe+fix. tee-only, no compiles until clippy.
+- 2026-05-30 AGENT 2 (config.rs): LanceConfig flusher cleanup. Found WritePath enum + write_path/flusher_tick_interval/disable_background_flusher already absent from current config.rs; LanceConfig already carried only `versioned`. Verified no WritePath refs/uses remain anywhere. Updated LanceConfig doc-comment to record the native rewrite drops those knobs (lance optimize owns compaction/GC). Left a // ///REVIEW: noting spec KEEP list mentions retention_ns but the struct never had it (env-var/background_optimizer owns retention) — did not invent a field. Touched ONLY the kv-lance LanceConfig region via tee; Memory/SurrealKv/RocksDb configs untouched.
+
+## 2026-05-30 — AGENT 1: native-only kvs/lance/mod.rs
+Rewrote `surrealdb/core/src/kvs/lance/mod.rs` to lance's NATIVE read/write path.
+commit() now builds one Arrow batch (writes + tombstone rows) and applies it with a
+single `MergeInsertBuilder::execute_reader` (moved `execute_merge` in from flusher.rs)
+= one commit / one lance version. Deleted commit_lsm/commit_legacy_gate and the
+memtable branch in get()/scan_impl(); reads are pending → lance (checkout_version via
+.ok() | latest) → filter/project/merge. Datastore now keeps {dataset, versioned,
+background_optimizer, commit_seq}; Transaction keeps {done, write, versioned, pending,
+save_points, read_version, dataset, background_optimizer, commit_seq}. Removed
+mod {commit_gate,flusher,memtable,wal} + their imports + WritePath. Kept proven helpers
+build_write_batch_lance/build_tombstone_batch_lance/max_persisted_seq/DatasetHandle and
+the test-only scan helpers. NOTE: tests.rs + integration_tests still reference removed
+items (WritePath, LanceConfig::{write_path,disable_background_flusher,flusher_tick_interval},
+commit_gate module) — they need agent 2/3 / orchestrator follow-up. Did NOT run cargo.
+
+## 2026-05-30T17:05 — AGENT 3: rewrote kvs/lance/tests.rs (Transactable contract only)
+**Scope:** surrealdb/core/src/kvs/lance/tests.rs (tee-only; no cargo run)
+- Kept 57 `#[tokio::test]` contract tests: lifecycle (kind/closed sticky/writeable/readonly+finished guards), get/exists (RYW, pending tombstone, committed, versioned-on-false errors), set/commit/cancel/overwrite/del round-trips, put/putc/delc CAS (match/mismatch/None-None/None-on-existing), scan/scanr/keys/keysr (order, half-open range, skip+limit, pending merge incl. override+delete-hide), ScanLimit Bytes/BytesOrCount, savepoints (rollback incl. tombstone restore, release, nested, empty-stack errors for both), versioning time-travel, concurrency (disjoint + same-key OCC), differential-vs-HashMap, optimizer-alive + shutdown-timeout, 3 timeline read-view tests.
+- Deleted all LSM/reinvention tests: writepath_*, lsm_recovery_*, seq_column_*, commit_gate_*, shutdown_drains_pending_commits, bench_lsm_*; dropped `WritePath` import + `scan_seqs/scan_versions/dataset_for_tests` usage.
+- LanceConfig field set ASSUMED = `{ versioned: bool }` only (matches the already-rewritten config.rs); every literal now sets only `versioned`.
+- 4 `// ///REVIEW:` anchors (all about "one commit = one lance version"): get_at_specific_version (checkout sees old-or-None), timeline versions_grow (≥2 lower-bound vs compaction), timeline view historical (v_after>v_before), timeline write+delete single-version (==before+1).
+
+## 2026-05-30T16:55 — agents 1+3 landed; orphans + integration_tests removed
+**Branch:** claude/sleepy-cori-aRK2x
+**Scope:** kvs/lance/{mod.rs(native),tests.rs(57 contract tests)}; deleted
+  memtable/wal/flusher/commit_gate/integration_tests.rs + WritePath.
+**Verdict:** IN PROGRESS (native source coherent; 8 // ///REVIEW anchors open)
+**Open:** REVIEW anchors — version stamp (read_version+1 vs latest+1),
+  lance OCC conflict -> Error::TransactionRetryable, get@version deletion-vector
+  semantics, timeline version-count assertions. Next: savant testers -> fix ->
+  strip /// -> clippy -> PR -> subscribe.
+
+## 2026-05-30T17:40 — 3 savants complete; consolidated fix plan
+**Verdict:** native rewrite compile+clippy-ready; 1 correctness BLOCKER + minors.
+**Fixes to apply (orchestrator):**
+- [BLOCKER] mod.rs execute_merge: `.map_err(|e| Error::Datastore(...))?` -> `.map_err(Error::from)?`
+  (reuse err.rs From<lance::Error> -> Error::TransactionConflict, the retryable variant).
+- [BLOCKER-2] err.rs From<lance::Error>: add arm for lance write-contention-exhaustion -> TransactionConflict.
+- [M1] reads @ latest for version=None = native lance-graph read-committed (INTENTIONAL) — document; reorder closed() before versioned check.
+- [doc] fix stale "arrow 57 / lance 4.0 / 1.0.4" -> "arrow 58 / lance 6.0.0"; fix read_version "dead_code" clause; drop stale mod.rs:1267 banner; soften schema "= Lance version" comment.
+- STRIP all `// ///REVIEW:` sentinels.
+**GATE COMMAND (Savant C — critical):** `cargo clippy -p surrealdb-core --features kv-lance --tests -- -D warnings`
+  (stock `cargo make ci-clippy` omits kv-lance and would cfg-strip the rewrite = false green).

.claude/board/EPIPHANIES.md

@@ -114,3 +114,118 @@ previously reclaimed that space immediately).
 
 **Cross-ref:** codex P1 on PR #29 (discussion_r3328296248); fix in this
 commit; regression `test_timeline_write_delete_commit_is_single_atomic_version`.
+
+## 2026-05-30 — No CI runs on this fork; .rustfmt.toml is split-brain (stable build, nightly-only fmt opts)
+**Status:** FINDING
+**Scope:** repo CI/tooling — `.github/workflows/ci.yml`, `.rustfmt.toml`, `rust-toolchain*`
+
+PR #29 head (5997eea) has ZERO check runs; the only commit status is the
+CodeRabbit review bot (pending). `ci.yml` triggers on every `pull_request`
+(no branch filter), so the absence is environmental: GitHub Actions is not
+enabled/approved on the AdaWorldAPI fork. Net: the only merge gate is the
+review bots + the human owner — there is no test/clippy/fmt enforcement.
+Separately, `.rustfmt.toml` enables nightly-only options (wrap_comments,
+imports_granularity=Module, group_imports=StdExternalCrate, comment_width)
+while the build toolchain is pinned stable 1.95 (`rust-toolchain.toml`); the
+fmt-only nightly (`rust-toolchain.nightly` = nightly-2025-08-07) is never run
+here. Running fmt under either stable or that nightly reformats the WHOLE
+crate (~1900 lines, 22+ files) => HEAD is not fmt-clean under its own config.
+Consequence: "future-proof" config that no gate enforces is pure drift. Per
+the org's 99%-stable policy (nightly only for Miri in ndarray), the resolution
+is to make the config stable-honest (comment out the unstable opts) and lean
+on stable tools (cargo-machete et al.). A one-time stable `cargo fmt`
+normalization is a separate, deliberate follow-up (not triggered here, to
+avoid mixing mass reformat churn into feature commits).
+
+**Cross-ref:** PR #29 check status (0 runs); ci.yml on-block; this commit's
+`.rustfmt.toml` change; GRIDLAKE_BUILD.md.
+
+## 2026-05-30 — Per-commit `seq` reset to 0 on every open → broke cross-restart monotonicity (savant BLOCKER, fixed)
+**Status:** FINDING
+**Scope:** `kvs/lance/mod.rs` (Datastore::new seeding), step-2 P3 seq column
+
+A 3-savant read-only review (Opus: concurrency/ACID, Lance data-model,
+idiom/tests) of step-2 passed 96/0 tests yet caught a BLOCKER tests
+structurally can't see: `commit_seq` was re-initialised to `AtomicU64::new(0)`
+on every `Datastore::new`, seeded only from the replayed (un-flushed) WAL
+record count, and NEVER advanced past the max `seq` already persisted in
+Lance — whereas the sibling `generation` counter IS advanced past the
+replayed-WAL max. After any restart where the WAL was truncated (data already
+flushed), new commits re-minted seq=1,2,3… colliding with / regressing below
+seqs already written to Lance, defeating the column's sole purpose (a globally
+monotonic, per-commit replay axis). Root distinction: `generation` is
+memtable-local and NOT persisted (need only clear the WAL tail); `seq` IS a
+Lance column, so it must seed from the persisted max. FIX:
+`Datastore::max_persisted_seq` (tolerant scan of the `seq` column; 0 for
+empty/legacy) seeds `commit_seq = AtomicU64::new(max_persisted_seq)` at open;
+replayed WAL records mint ABOVE that floor. Regression
+`seq_survives_restart_above_persisted_max` fails on the old code, passes on the
+fix. Verified: 98 kvs::lance tests pass, 0 fail.
+
+Documented (accepted, pre-release) limitations the savants surfaced:
+- No schema migration for a pre-`seq` (4-col) on-disk dataset (fresh only today).
+- WAL carries no persisted seq → replay renumbers seqs above the persisted max
+  (monotonic+unique), not to exact pre-crash values.
+- LegacyCommitGate stamps `seq = version` (per-commit fidelity is LSM-only).
+- `seq`/`generation` mint from two atomics → under concurrency seq order can
+  disagree with commit order (harmless today; reads never consult seq).
+
+**Cross-ref:** `.claude/board/GRIDLAKE_REVIEW.md` (S1/S2/S3); fix in this commit.
+
+## 2026-05-30 — kv-lance substrate maps onto Cognitive-RISC invariants; do NOT add class_id to it
+**Status:** FINDING
+**Scope:** `kvs/lance/*` vs `lance-graph/.claude/specs/{cognitive-risc-core,cognitive-risc-classes,wikidata-hhtl-load,faiss-homology-cam-pq}.md`
+
+The kv-lance backend IS the "Substrate" layer (row 1) of the Cognitive-RISC
+five-layer stack ("SoA, LE byte contract, surrealkv WAL/ACID, policy-free
+state"). Concrete mapping: CommitGate/single_lance_commit = the sole cold-path
+writer (invariant #4); WAL+memtable ↔ flusher→Lance two-clock decoupling +
+the adaptive-batching rate-floor = the shock absorber (#7); WAL carries KV
+rows only, never compiled candidates (#11); the schema is opaque (key,val) +
+MVCC bookkeeping version/tombstone/seq with ZERO domain meaning (#1, and #6
+permits generation/tombstone counters). The step-2 `seq_survives_restart`
+test is exactly the spec's "smallest first slice" (WAL round-trip + read back
+after a simulated restart).
+
+TRAP recorded so a future session does not weld the inversion shut: freeze-
+time move **N1 ("add class_id/shape_id to the SoA")** must NOT be applied to
+the kv-lance schema — that violates invariant #1. class_id, HHTL nibble-path,
+facet bitmasks, and the CAM (BLAKE) hash live ONE LAYER UP (inside the `val`
+payload or lance-graph's own Lance datasets), never as kv-lance columns. The
+minimal key/val/version/tombstone/seq schema is correct precisely because it
+is policy-free.
+
+Live fork for this work — **F2**: spec default-leans "federate via DataFusion
+catalog (Arrow TableProviders)", not "read Lance directly (heavy/fragile)".
+kv-lance is the direct path; the step-1 Timeline ("SurrealDB-as-view-over-
+Lance", Rubicon) is the federation-shaped read surface. Decide: SurrealDB as
+writer-of-record into Lance (kv-lance) vs DataFusion-federated view (F2); they
+can coexist but the version-coupling risk is real. Version pin skew: repo on
+lance =6.0.0/arrow 58; spec pins lance 6.0.1/lancedb 0.29/datafusion 53.
+
+**Cross-ref:** PR #29/#30; lance-graph .claude/specs/ (sha d1635db).
+
+## 2026-05-30 — Audit: GRIDLAKE §8 Phase 1 & Phase 2 are IMPLEMENTED, not ROADMAP
+**Status:** FINDING
+**Scope:** surrealdb/core/src/kvs/lance/{flusher,schema,mod}.rs vs .claude/lance-backend/GRIDLAKE.md §8
+
+A fresh-session audit of the tree on `claude/sleepy-cori-aRK2x` (HEAD 55fc45c)
+against the GRIDLAKE §8 phased roadmap finds the §8 tags stale: Phase 1
+(adaptive batching) and Phase 2 (per-row seq) are both shipped + tested while
+§8 still reads ROADMAP / IN PROGRESS. Evidence — Phase 1: `FlusherConfig`
+carries the byte trigger `max_pending_bytes` (flusher.rs:74, default 8 MiB)
+and the rate floor `min_flush_interval` (flusher.rs:79, default 50 ms);
+`should_flush` (flusher.rs:271-285) gates trigger-driven + periodic flushes on
+the floor; locked by `should_flush_triggers_on_bytes` (flusher.rs:441) and
+`should_flush_respects_rate_floor` (flusher.rs:460). Phase 2: `seq: UInt64` is
+the 5th schema column (schema.rs:62), threaded through both batch builders with
+parallel-length assertions (mod.rs:1180 build_write_batch_lance, mod.rs:1242
+build_tombstone_batch_lance); `commit_seq` (mod.rs:177) seeds from the on-disk
+max via `max_persisted_seq` (mod.rs:203-239) — the savant BLOCKER fix — and
+55fc45c fails fast on a legacy 4-column dataset. `cargo check -p surrealdb-core
+--features kv-lance --tests` → 0 errors, 6 dead-code warnings (unwired
+TimelineView consumer); prior run 98 kvs::lance tests pass. §8 should read
+IMPLEMENTED for Phases 1-2. Per the append-only discipline the GRIDLAKE doc
+body was NOT mutated; this entry is the canonical re-tag record.
+
+**Cross-ref:** commits 7266acf, e329a7a, 55fc45c; GRIDLAKE.md §8; this session's grep/check audit.