feat(knowledge): put the vector store in Postgres, off the disk that gets wiped

CHANGELOG.md

@@ -6,6 +6,64 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Fixed — the vector store lived on a disk that is wiped on every restart
+
+`CHROMA_PERSIST_DIR` is `/app/data/chroma`: the Heroku container filesystem. It does
+not survive a dyno restart, and `web` and `worker` are separate process types with
+separate copies of it. Production, 2026-08-27, five hours after a deploy:
+
+    22:00:44  repair_embeddings  "Vector store empty but 758 docs in DB.
+                                  Forcing a full re-index."
+    22:01:21  code_symbol_embed  started   (all 8 646 files — force_full)
+    22:39:09  code_symbol_embed  completed (38 min)
+    23:59:50  heartbeat stops — the 7 200 s ceiling, at 380 of 758 documents
+    00:04     stale run reaped
+
+A full rebuild costs 12 039 s and the nightly budget is 7 200 s, so the store was
+empty again the next night, and the night after. **`index_repo` has completed 16
+times in 94 runs; `daily_sync` 13 in 91.** The self-repair (C3, v1.13.0) was not
+wrong — the repair cost more than the budget allowed, which no ceiling could fix.
+
+`PgVectorStore` puts the vectors in Postgres instead: one store, shared by every
+dyno, untouched by a restart. It is a drop-in — same six methods, same shapes, same
+cosine distance, and the collection handle still answers `count()`, because that is
+what `pipeline_runner.py:415` reads to decide whether a full re-index is owed.
+
+**Embeddings do not change.** ChromaDB computed them internally with its bundled ONNX
+`all-MiniLM-L6-v2`; this calls the same class directly. Identical vectors, identical
+384 dimensions, and `vector_cosine_ops` because the collections it replaces were
+created with `{"hnsw:space": "cosine"}` — so a query returns the neighbours it
+returned before. Verified end to end against the production database: the same four
+documents, the same ranking, distances 0.2234 / 0.7867 / 1.0161.
+
+HNSW rather than IVFFlat: it needs no training pass, so it is correct from an empty
+table — IVFFlat built on an empty table returns nothing until rebuilt, which is the
+exact failure this change exists to remove. Measured on the schema with 30 000 rows
+of 384 dimensions: build 20 s, and a top-5 filtered by `project_id` goes from
+4 738 ms with no vector index to 0.92 ms with one. `hnsw.iterative_scan` was measured
+too and makes no difference at this selectivity — the 65 ms it appeared to save was a
+cold cache.
+
+`VECTOR_STORE_BACKEND` selects the backend and still defaults to `chroma`, so the
+flip is a decision taken on a verified deployment rather than a side effect of this
+merge. psycopg sits beside asyncpg deliberately: the interface is called
+synchronously from ten sites including the agent's hot path, and converting those to
+async is a larger, riskier change than carrying a second driver.
+
+pgvector is available on **both** deployments — 0.8.1 on Heroku Postgres, 0.8.2 on
+Supabase — so this fix does not have to wait for a database move.
+
+### Added — a migration graph that says when it is broken
+
+`b1c2d3e4f5a6` was already taken. Alembic does not say so: it reports
+`CycleDetected` naming four unrelated revisions, and only when something walks the
+graph — so a migration with a duplicate id can be committed and merged first. Revision
+ids here are hand-picked rather than generated, which is what puts that within reach.
+Three checks now read the graph with `ast` (a first attempt used a regex and reported
+seven heads where alembic reports one, because merge migrations name their parents as
+a tuple), plus a fourth that fails if the tuple form ever stops being understood.
+
+
 ### Fixed — a full re-index could not finish, and the ceiling was the symptom
 
 Three ceilings cut three runs off on 2026-08-27 — 1800.02 s inside `code_symbol_embed`,

CLAUDE.md

@@ -323,7 +323,7 @@ Learnings are stored per-connection by default (`cross_connection_learnings_enab
 ### Storage
 
 - App data: SQLite in dev (`backend/data/agent.db`), PostgreSQL in production (`DATABASE_URL`).
-- Vectors: ChromaDB (`CHROMA_PERSIST_DIR` or `CHROMA_SERVER_URL` for remote); collections named `project_{project_id}`.
+- Vectors: **`VECTOR_STORE_BACKEND` picks the backend** — `pgvector` (Postgres table `doc_embeddings`, one row per chunk, HNSW `vector_cosine_ops`) or `chroma` (the default, `CHROMA_PERSIST_DIR` or `CHROMA_SERVER_URL`; collections named `project_{project_id}`). **ChromaDB's persist dir on Heroku is the container filesystem** — wiped on every dyno restart, and `web`/`worker` are separate process types with separate copies. An empty store makes `pipeline_runner` set `force_full`, a full rebuild costs 12 039 s against the nightly ceiling of 7 200 s, so the store was empty again by morning: **`index_repo` completed 16 times in 94 runs**. Embeddings are identical across backends (bundled ONNX `all-MiniLM-L6-v2`, 384-d) and the metric matches the `{"hnsw:space": "cosine"}` the collections were created with, so the swap does not move retrieval ranking. pgvector is available on both deployments (0.8.1 Heroku, 0.8.2 Supabase). Requires Postgres — the migration is a deliberate no-op on SQLite, and asking for pgvector there fails at start-up saying so.
 - BM25 snapshots: `backend/data/bm25/{project_id}.json.gz` and `schema_{connection_id}.json.gz` — **gzip JSON, not pickle, since 2026-08-21 (F-KNOW-06)**: `pickle.load` executes its payload, and `BM25_DATA_DIR` is configurable. The tokenized corpus is stored and `BM25Okapi` is rebuilt on load; a leftover `.pkl` is deleted, never read. Both are rebuilt from Postgres at start-up when missing (`app/ops/bm25_local_reconcile.py`).
 - Redis (`REDIS_URL`): rate limiting, agent concurrency tokens, WS tickets, ARQ task queue. In-memory fallback for dev — keep it working when adding Redis features.
 - Backups: `backend/data/backups/` when `backup_enabled=True`.

backend/.env.example

@@ -166,6 +166,20 @@ CORS_ORIGINS=["http://localhost:3000","http://localhost:3100","https://checkmyda
 # AUTO_INVESTIGATE_BUDGET_ENFORCEMENT_ENABLED=true  # skip auto-investigation when the owner is over budget or unresolved; false to always allow
 
 # ----- ChromaDB vector store ---------------------------------------------------
+# Which backend holds the vectors: "chroma" or "pgvector".
+#
+# "chroma" writes to CHROMA_PERSIST_DIR. On Heroku that is the container filesystem,
+# wiped on every dyno restart and NOT shared between the `web` and `worker` process
+# types. Measured 2026-08-27: the worker found the store empty five hours after a
+# deploy, so the pipeline forced a full re-index; a full rebuild costs 12 039 s and
+# the nightly ceiling is 7 200 s, so the run was reaped and the store was empty again
+# the next night. 16 of 94 repo-index runs have ever completed.
+#
+# "pgvector" puts them in Postgres — one store, every dyno, surviving restarts.
+# Requires a PostgreSQL DATABASE_URL: on SQLite the doc_embeddings migration is a
+# deliberate no-op, and asking for pgvector there fails at start-up saying so.
+# Embeddings are unchanged either way (ONNX all-MiniLM-L6-v2, 384-d).
+# VECTOR_STORE_BACKEND=chroma
 # CHROMA_PERSIST_DIR=./data/chroma
 # Remote ChromaDB. Set this to move the vector index OUT of the app process: with it
 # unset, an embedded PersistentClient keeps the HNSW graph in memory and the files on

backend/alembic/versions/1d72054cd637_doc_embeddings_pgvector.py

@@ -0,0 +1,91 @@
+"""doc_embeddings: move the vector store off local disk and into Postgres
+
+Revision ID: 1d72054cd637
+Revises: 6287a47828ca
+Create Date: 2026-08-28
+
+Why this table exists is in ``app/models/doc_embedding.py``. The short version:
+ChromaDB persisted to the dyno's container filesystem, which is wiped on every
+restart and is not shared between the ``web`` and ``worker`` process types. An empty
+store makes the pipeline force a full re-index, a full re-index of the one real
+customer repository costs 12 039 s, and the nightly job's ceiling is 7 200 s — so
+16 of 94 repo-index runs ever completed and the store was empty again by morning.
+
+**Postgres only, deliberately.** Development and the test suite run on SQLite
+(``backend/data/agent.db``), where there is no ``vector`` extension and no HNSW.
+``make setup`` runs ``alembic upgrade head`` against that SQLite file, so a migration
+that assumed Postgres would break every developer's first command. On SQLite this
+migration is a no-op and the ChromaDB backend stays in use; ``VECTOR_STORE_BACKEND``
+selects between them.
+
+Measured before choosing HNSW over IVFFlat, on this exact schema with 30 000 rows of
+384 dimensions: build 20 s, and a top-5 filtered by ``project_id`` goes from 4 738 ms
+with no vector index to 0.92 ms with one. HNSW also needs no training pass, so it is
+correct from an empty table — IVFFlat built on an empty table returns nothing until
+it is rebuilt, which is exactly the failure mode this whole table exists to remove.
+
+pgvector is available on both deployments: 0.8.1 on Heroku Postgres, 0.8.2 on
+Supabase. Checked 2026-08-28 — this migration is therefore safe to run before any
+database move, which is why the vector fix does not have to wait for one.
+"""
+
+from alembic import op
+
+revision = "1d72054cd637"
+down_revision = "6287a47828ca"
+branch_labels = None
+depends_on = None
+
+
+def _is_postgres() -> bool:
+    return op.get_bind().dialect.name == "postgresql"
+
+
+def upgrade() -> None:
+    if not _is_postgres():
+        return
+
+    op.execute("CREATE EXTENSION IF NOT EXISTS vector")
+
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS doc_embeddings (
+            project_id  TEXT        NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+            id          VARCHAR(512) NOT NULL,
+            document    TEXT        NOT NULL,
+            metadata    JSONB       NOT NULL DEFAULT '{}'::jsonb,
+            embedding   vector(384) NOT NULL,
+            updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
+            PRIMARY KEY (project_id, id)
+        )
+        """
+    )
+
+    # Equality on the one metadata key that is ever queried alone — the path that
+    # removes a changed file's chunks before re-embedding them.
+    op.execute(
+        """
+        CREATE INDEX IF NOT EXISTS ix_doc_embeddings_source_path
+        ON doc_embeddings (project_id, (metadata ->> 'source_path'))
+        """
+    )
+
+    # `vector_cosine_ops` because the ChromaDB collections this replaces were created
+    # with `{"hnsw:space": "cosine"}` — so `<=>` returns the same distance the old
+    # store returned, and retrieval ranking does not silently change under the fix.
+    op.execute(
+        """
+        CREATE INDEX IF NOT EXISTS ix_doc_embeddings_hnsw
+        ON doc_embeddings USING hnsw (embedding vector_cosine_ops)
+        """
+    )
+
+
+def downgrade() -> None:
+    if not _is_postgres():
+        return
+    op.execute("DROP INDEX IF EXISTS ix_doc_embeddings_hnsw")
+    op.execute("DROP INDEX IF EXISTS ix_doc_embeddings_source_path")
+    op.execute("DROP TABLE IF EXISTS doc_embeddings")
+    # The extension is deliberately NOT dropped: another table may come to depend on
+    # it, and dropping an extension takes its types with it.

backend/app/agents/knowledge_agent.py

@@ -23,7 +23,7 @@
 from app.knowledge.entity_extractor import ProjectKnowledge
 from app.knowledge.hybrid_retriever import HybridRetriever
 from app.knowledge.reranker import build_reranker
-from app.knowledge.vector_store import VectorStore
+from app.knowledge.vector_store import VectorStore, make_vector_store
 from app.llm.base import LLMResponse, Message, ToolCall
 from app.services.project_cache_service import ProjectCacheService
 
@@ -47,7 +47,7 @@ def __init__(
         vector_store: VectorStore | None = None,
         hybrid_retriever: HybridRetriever | None = None,
     ) -> None:
-        self._vector_store = vector_store or VectorStore()
+        self._vector_store = vector_store or make_vector_store()
         # The hybrid retriever (M3) is constructed lazily so that disabling the
         # feature flag carries zero startup cost. When the flag flips on, a
         # single instance is shared across requests.

backend/app/agents/orchestrator.py

@@ -69,7 +69,7 @@
 from app.core.workflow_tracker import tracker as default_tracker
 from app.knowledge.custom_rules import CustomRulesEngine
 from app.knowledge.repo_analyzer import RepoAnalyzer
-from app.knowledge.vector_store import VectorStore
+from app.knowledge.vector_store import VectorStore, make_vector_store
 from app.llm.base import LLMResponse, Message, ToolCall
 from app.llm.errors import (
     RETRYABLE_LLM_ERRORS,
@@ -362,7 +362,7 @@ def __init__(
         mcp_source_agent: MCPSourceAgent | None = None,
     ) -> None:
         self._llm = llm_router or LLMRouter()
-        self._vector_store = vector_store or VectorStore()
+        self._vector_store = vector_store or make_vector_store()
         self._custom_rules = custom_rules or CustomRulesEngine()
         self._tracker = workflow_tracker or default_tracker
         self._validator = AgentResultValidator()

backend/app/agents/sql_agent.py

@@ -44,7 +44,7 @@
 from app.core.validation_loop import ValidationLoop
 from app.knowledge.custom_rules import CustomRulesEngine
 from app.knowledge.entity_extractor import ProjectKnowledge
-from app.knowledge.vector_store import VectorStore
+from app.knowledge.vector_store import VectorStore, make_vector_store
 from app.llm.base import LLMResponse, Message, ToolCall
 from app.llm.retry import llm_call_with_retry
 from app.llm.router import LLMRouter
@@ -125,7 +125,7 @@ def __init__(
         rules_engine: CustomRulesEngine | None = None,
     ) -> None:
         self._llm = llm_router or LLMRouter()
-        self._vector_store = vector_store or VectorStore()
+        self._vector_store = vector_store or make_vector_store()
         self._rules_engine = rules_engine or CustomRulesEngine()
         self._cache_svc = ProjectCacheService()
 

backend/app/api/routes/repos.py

@@ -24,7 +24,7 @@
 from app.knowledge.pipeline_runner import IndexingPipelineRunner
 from app.knowledge.repo_analyzer import RepoAnalyzer
 from app.knowledge.repo_url import validate_git_ref, validate_repo_url
-from app.knowledge.vector_store import VectorStore
+from app.knowledge.vector_store import make_vector_store
 from app.models.base import async_session_factory
 from app.services.checkpoint_service import CheckpointService
 from app.services.connection_service import ConnectionService
@@ -43,7 +43,7 @@
 _repo_analyzer = RepoAnalyzer(settings.repo_clone_base_dir)
 _doc_store = DocStore()
 _doc_generator = DocGenerator()
-_vector_store = VectorStore()
+_vector_store = make_vector_store()
 _cache_svc = ProjectCacheService()
 _checkpoint_svc = CheckpointService()
 

backend/app/config.py

@@ -94,6 +94,23 @@ def _fix_database_url(self) -> "Settings":
     #: new key; once its pending count reaches zero the old key can be dropped.
     master_encryption_keys_old: str = ""
 
+    # F-KNOW-13: which backend holds the vectors.
+    #
+    # "chroma" writes to `chroma_persist_dir`, which on Heroku is the container
+    # filesystem — wiped on every dyno restart, and NOT shared between the `web` and
+    # `worker` process types. Measured 2026-08-27: the worker found the store empty
+    # five hours after a deploy, so the pipeline set `force_full`; a full rebuild of
+    # the one real customer repository costs 12 039 s against the nightly ceiling of
+    # 7 200 s; the run was reaped and the store was empty again the next night.
+    # 16 of 94 repo-index runs have ever completed.
+    #
+    # "pgvector" puts them in Postgres, which every dyno shares and a restart does
+    # not touch. Requires Postgres — development and the test suite run on SQLite,
+    # where the migration is a no-op and this must stay "chroma".
+    #
+    # Default is still "chroma" so the switch is a deliberate flip on a verified
+    # deployment rather than a side effect of deploying this change.
+    vector_store_backend: str = "chroma"
     chroma_persist_dir: str = "./data/chroma"
     chroma_server_url: str = ""
     chroma_embedding_model: str = Field(