fix(indexing): lancedb migration guards + E10 negative result

[ManSio]

What

Critical production fix for silent lancedb schema loss + a recorded negative experiment.

1. fix(indexing): guard lancedb migration + recreate against silent schema loss

   • db_manager imported IndexerTableMixin class methods as module-level functions -> ImportError -> column migration ran silently nowhere (prod incident).
   • db_writer.is_table_missing treated field 'X' does not exist in table schema (schema-mismatch) as missing table -> full rebuild (drop + ~13 min re-embed) instead of soft migration.
   • Now: migrations implemented locally with pa.field from self.schema; recreate only on real table absence ("in table schema" excluded). +200 lines of regression tests (tests/test_lancedb_recreate.py, 12 passed).

2. refactor(settings): restore MAX_RERANKER_INPUT default 30 behind env toggle — behavior identical to HEAD; pool re-test possible via env.

3. chore(experiments): keep scripts/e2e_quality_search.py (quality harness) + experiments/search_quality/E10_full_text_embed.py (rerunnable artifact).

4. docs: migration incident + E10 negative result (EXPERIMENTS_LOG / AGENT_DIARY / KNOWN_ISSUES / portfolio exp-43).

Experiment E10 (REFUTED)

Hypothesis: full-text chunk embedding + e5 query:/passage: prefixes (llama.cpp branch) + reranker pool 50 lift search hit. Clean rebuild: 599 files / 9514 chunks in 799.9s. Result vs author baseline — within N=10 noise: fast hit@1=0%/- hit@5=50%/50%; quality hit@1=20%(baseline 30%) hit@5=40%(baseline 30%). Code reverted to HEAD. Pure-vector plateau reconfirmed; next: AST/Graph-hybrid re-ranking.

Verification

• 1756 passed / 5 skipped full suite (verify_diary pre-commit gate).
• ruff clean; all 9 pre-commit gates green on every commit.
• Portfolio guard tests (lab.test.ts + evidence-eval.test.ts): 26/26 passed (exp-43 EN/RU congruent).
• ⚠️ verified_from_clean_state not run locally (all full-suite gates passed in pre-commit); migration live-check on existing prod DB is a follow-up (owner question).

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: c8ec35b4-8cd0-4175-a79a-c41baa8b70d4

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}