fix(memory): paginate memory lifecycle sweeps and stop dropping rows past the first page (salvage #772 + #776)

[unohee]

Salvages the memory-lifecycle work from draft PRs #772 and #776 into one coherent change. Both drafts were rewritten from a stale base; only the substantive hunks were lifted onto current origin/main, with main's API surface preserved.

What was kept

Pagination (from #772) — every memory-lifecycle sweep read the table with a single capped query, so larger stores were silently truncated:

Function	Before	After
compactMemoryTable	.limit(100_000) + hard refusal at the limit	offset/limit pages via table.query()
cleanupExpired	.limit(10_000)	full table via fetchAllTableRows
consolidateMemories	.limit(10_000) vector query	full table via fetchAllTableRows
getMemoryStats	.limit(10_000) vector query	full table via fetchAllTableRows

compactMemoryTable's read path no longer refuses at a safety limit, because it no longer loads the whole table in one query.

shouldCompact (from #772) — uses countRows() for a bounded total and estimates waste from a single sampled page instead of loading the table; legacy v2 columns are now detected from the table schema rather than from row values, where they were only ever visible on v2 rows.

Bucket dedup (from #776) — removeDuplicates previously did an O(n²) pairwise scan. It now buckets records by a stable FNV-1a hash of their identity fields (repo, type, derivedFrom, canonical metadata) and compares vectors only within a bucket. This is what makes cross-page deduplication correct: with the old pairwise scan over page-filtered input, a duplicate pair split across a page boundary could never be compared. Within a bucket, records are ranked by importance then recency, so the surviving record does not depend on the order pages were read in.

Type defect in #772 fixed — cosineSimilarity was declared : boolean while returning a similarity score, breaking the >= CONSOLIDATION_SIMILARITY comparison at its call site (TS2365 + TS2322 on the draft head). It is correctly typed : number here.

What was dropped

Junk / scratch — package.json + package-lock.json version churn (0.22.1 → 0.23.0, stale against main's 0.24.3); no dependency changed, so no lockfile regeneration was needed.

Dead code / wrong API (from #776's head) — getTable(tempTableName) (the real getTable() takes no argument), db.renameTable() (not on @lancedb/lancedb's Connection), and its rewrite of the table-swap path.

Main's API surface preserved (reverted from #776's head) — the deletions of withMemoryMutationLock, saveConversation, getRecentConversations, runBackgroundCognition and getMemoryStats. All are still exported by main and still called by discordCore.ts, support/chatMemory.ts and support/web.ts; the draft head would not have compiled.

Main's hardening kept — compactMemoryTable still rethrows on failure (the draft's unconditional catch → return {0,0,0,0} would have swallowed real errors), and consolidateMemories still runs under withMemoryMutationLock (the draft wrapped it in withMemoryWriteRetry, which deadlocks here — the updateMemoryRecord/deleteMemoryIds calls inside it each acquire withMemoryWriteLock).

Verification

• npx tsc --noEmit — clean.
• npx vitest run src/memory/compaction.test.ts src/memory/memoryOps.test.ts src/core/service.test.ts — 57 passed (src/core/service.test.ts exercises the compactMemoryTable/shouldCompact/cleanupBackupFiles consumers). Pre-commit oxlint (91 rules) and type check passed.

Tests added

• compaction.test.ts: a straddling-pair case — a 20,001-record input whose near-duplicate pair sits across the 10,000-record page boundary — plus a case asserting records that share identity metadata but have dissimilar vectors are not merged.

An additional throwaway smoke test (not committed) exercised all four sweeps against a 25k-row fake table to confirm each now sees every row and merges the cross-page pair; it was removed after passing.

Closes the salvage of #772 and #776.