Enforce Embedding.embedder_path NOT NULL to match partial unique constraints

The field was declared null=True, blank=True while its Python annotation
claimed str, surfacing as a mypy assignment mismatch in the #1331 baseline.
More importantly, the unique_embedding_per_{document,annotation,note,
conversation,message}_embedder partial unique constraints (migration 0059)
key on (embedder_path, <parent>) and filter by <parent>__isnull=False — so
any row with NULL embedder_path bypassed duplicate prevention for its
parent entirely.

Every production creation path already supplies a concrete embedder_path:
Embedding.objects.store_embedding(), HasEmbeddingMixin.add_embedding(),
worker_uploads._store_embeddings() (which skips when empty). Enforcing
NOT NULL matches actual behaviour.

Migration 0068 backfills legacy NULL rows with settings.DEFAULT_EMBEDDER,
deleting rows whose backfill would collide with an existing
(default_embedder_path, parent) row (those rows were already unreachable
since no query path filters on NULL). Runs atomic=False so the RunPython
commits before AlterField takes the ACCESS EXCLUSIVE lock to SET NOT NULL,
matching the pattern from migration 0059.

Also removed the now-unreachable `or 'Unknown Model'` fallback in
Embedding.__str__.

Closes #1357

Author: claude

CHANGELOG.md

@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **`Embedding.embedder_path` could be NULL but was typed `str`** (Issue #1357, `opencontractserver/annotations/models.py:461-465`, `opencontractserver/annotations/models.py:584-585`, `opencontractserver/annotations/migrations/0068_enforce_embedder_path_not_null.py`): The Django field was declared `null=True, blank=True` while the Python annotation claimed `str`, causing a long-standing mypy `assignment` error and — more importantly — silently gutting the partial unique constraints added in migration 0059. Each `unique_embedding_per_{document,annotation,note,conversation,message}_embedder` constraint is conditioned on `<parent>__isnull=False` and keys on `(embedder_path, <parent>)`, so any row with `embedder_path IS NULL` bypassed duplicate prevention for its parent. Every production code path that creates an `Embedding` (`Embedding.objects.store_embedding()`, `HasEmbeddingMixin.add_embedding()`, `worker_uploads._store_embeddings()`) already supplies a concrete `embedder_path` or skips creation when empty, so enforcing non-null at the DB level matches actual behaviour rather than constraining it. New migration 0068 backfills any legacy NULL rows with `settings.DEFAULT_EMBEDDER` (deleting rows that would collide with an existing `(default_embedder_path, parent)` row under the partial unique constraint — they were previously unreachable via any query path since all call sites filter on a concrete embedder path), then `AlterField`s the column to `NOT NULL`. Removed the now-unreachable `or 'Unknown Model'` fallback in `Embedding.__str__`. Migration runs with `atomic = False` so the RunPython backfill commits before `AlterField` takes the `ACCESS EXCLUSIVE` lock to set `NOT NULL`, matching the pattern established by migration 0059.
+
 ### Added
 
 - **Mypy type-checking wired into pre-commit and CI** (Issue #1331): The existing `[mypy]` block in `setup.cfg` and the `mypy==1.20.1` / `django-stubs==6.0.2` / `djangorestframework-stubs==3.16.9` pins in `requirements/local.txt` were never actually enforced, so the investment was drifting (48 pre-existing `# type: ignore` markers, many modules at 0% annotation coverage). This PR turns on the gate without requiring the 7208 existing errors across 357 files to be fixed first — `mypy.ini` lists each of those files under its own `[mypy-<module>] ignore_errors = True` section, so new modules added outside the baseline **are** type-checked and CI / the hook fails on their errors.

opencontractserver/annotations/migrations/0068_enforce_embedder_path_not_null.py

@@ -0,0 +1,112 @@
+"""
+Backfill any legacy Embedding rows with NULL ``embedder_path`` and then
+tighten the column to ``NOT NULL``.
+
+Context (issue #1357): ``Embedding.embedder_path`` was declared
+``null=True, blank=True`` on the Django field while its Python annotation
+claimed ``str``. The partial unique constraints added in migration 0059
+reference ``embedder_path`` with ``condition=Q(<parent>__isnull=False)``,
+meaning any row where ``embedder_path`` is NULL silently bypasses duplicate
+prevention. Every production creation path (store_embedding, add_embedding,
+worker_uploads) already supplies a concrete value, so we enforce the
+invariant at the DB level to match.
+
+Backfill strategy:
+  1. For each NULL-embedder_path row, set ``embedder_path`` to
+     ``settings.DEFAULT_EMBEDDER``.
+  2. If that assignment would collide with an existing (embedder_path,
+     parent) row under a partial unique constraint, delete the NULL row
+     instead — it cannot be matched by any query (all call sites filter
+     on a concrete ``embedder_path``) so it was effectively dead data.
+"""
+
+import logging
+
+from django.conf import settings
+from django.db import IntegrityError, migrations, models, transaction
+
+logger = logging.getLogger(__name__)
+
+
+def backfill_null_embedder_paths(apps, schema_editor):
+    Embedding = apps.get_model("annotations", "Embedding")
+
+    default_embedder_path = getattr(settings, "DEFAULT_EMBEDDER", "") or ""
+    if not default_embedder_path:
+        logger.warning(
+            "settings.DEFAULT_EMBEDDER is empty; NULL embedder_path rows will "
+            "be deleted because no sensible backfill value is available."
+        )
+
+    null_rows = Embedding.objects.filter(embedder_path__isnull=True)
+    total = null_rows.count()
+    if total == 0:
+        logger.info("No Embedding rows with NULL embedder_path — nothing to backfill.")
+        return
+
+    backfilled = 0
+    deleted = 0
+
+    # Use .iterator() to avoid loading the full set into memory on large tables.
+    for emb in null_rows.iterator(chunk_size=500):
+        if not default_embedder_path:
+            emb.delete()
+            deleted += 1
+            continue
+
+        emb.embedder_path = default_embedder_path
+        try:
+            with transaction.atomic():
+                emb.save(update_fields=["embedder_path"])
+            backfilled += 1
+        except IntegrityError:
+            # A (default_embedder_path, parent) row already exists and is
+            # covered by the partial unique constraint. The legacy NULL row
+            # cannot be queried (no call site filters on NULL), so dropping
+            # it is a lossless cleanup.
+            logger.info(
+                "Dropping NULL-embedder_path Embedding id=%s: backfill to %r "
+                "would duplicate an existing row under the partial unique "
+                "constraint.",
+                emb.pk,
+                default_embedder_path,
+            )
+            emb.delete()
+            deleted += 1
+
+    logger.warning(
+        "Embedding.embedder_path backfill complete: backfilled=%s, deleted=%s, "
+        "total=%s.",
+        backfilled,
+        deleted,
+        total,
+    )
+
+
+def reverse_backfill(apps, schema_editor):
+    """No-op: we cannot restore rows that were deleted, and re-nulling
+    backfilled rows would be indistinguishable from values that have always
+    been ``settings.DEFAULT_EMBEDDER``."""
+
+
+class Migration(migrations.Migration):
+    atomic = False
+
+    dependencies = [
+        ("annotations", "0067_merge_20260316_0312"),
+    ]
+
+    operations = [
+        migrations.RunPython(backfill_null_embedder_paths, reverse_backfill),
+        migrations.AlterField(
+            model_name="embedding",
+            name="embedder_path",
+            field=models.CharField(
+                help_text=(
+                    "Identifier for the embedding model or pipeline used "
+                    "(e.g. 'openai/text-embedding-ada-002')."
+                ),
+                max_length=256,
+            ),
+        ),
+    ]

opencontractserver/annotations/models.py

@@ -458,11 +458,9 @@ class Embedding(BaseOCModel):
         help_text="References the ChatMessage that this embedding belongs to (if any).",
     )
 
-    # The name/path of the model used to generate this embedding
+    # Required: NULL would silently bypass the partial unique constraints below.
     embedder_path: str = django.db.models.CharField(
         max_length=256,
-        null=True,
-        blank=True,
         help_text="Identifier for the embedding model or pipeline used (e.g. 'openai/text-embedding-ada-002').",
     )
 
@@ -584,7 +582,7 @@ class Meta:
         verbose_name_plural = "Embeddings"
 
     def __str__(self):
-        return f"Embedding (ID={self.pk}) [{self.embedder_path or 'Unknown Model'}]"
+        return f"Embedding (ID={self.pk}) [{self.embedder_path}]"
 
 
 class StructuralAnnotationSet(BaseOCModel):