Summary
cluster_vocab_<model_tag>.npz files are written per embedding model and never pruned. Every model a user configures leaves a permanent ~5 MB file in the databases directory, including models they have since removed.
This was harmless until #111, because a filename bug meant the cache never actually persisted. Now that it does, the files accumulate for real.
Where
invokeai/app/services/image_index/image_index_default.py::_build_vocab_embeddings writes:
model_tag = self._model_id.replace(":", "_")[:24]
cache_path = self._invoker.services.configuration.db_path.parent / f"cluster_vocab_{model_tag}.npz"start() already prunes the database side for models that are no longer current — delete_embeddings_for_other_models — but there is no equivalent sweep for these files. Nothing globs the databases directory (the only other consumers, sqlite_migrator_impl.py and sqlite_database.py, address files by exact name), so the strays are inert; the cost is disk only.
Size
Real file from a live install: cluster_vocab_blake3_613228c1fd904fb8e.npz, 5.1 MB — a (1675, 768) float32 matrix plus the fingerprint. So roughly 5 MB per embedding model ever configured, forever.
Reproduction
- Run with
image_index_model set to a CLIP Vision model; let the vocabulary build. A cluster_vocab_<tagA>.npz appears.
- Change
image_index_model to a different model and restart. A second cluster_vocab_<tagB>.npz appears.
- Uninstall the first model.
Observed: cluster_vocab_<tagA>.npz remains indefinitely.
Expected: it is removed alongside the embeddings that were dropped for the same model.
Suggested direction
Extend the existing start() cleanup that calls delete_embeddings_for_other_models to also glob cluster_vocab_*.npz in db_path.parent and unlink any whose tag is not the current model's. Worth doing as a best-effort pass wrapped in a handler — a failure to delete a stale cache should never prevent startup.
Two adjacent details for whoever picks this up: the [:24] truncation makes the tag lossy (fine for collision purposes — ≥17 hex chars survive — but the sweep must compare truncated tags, not full ids), and any glob should be careful not to delete a .tmp.npz staging file belonging to a concurrently-running instance. See #121 on staging-file naming.
Summary
cluster_vocab_<model_tag>.npzfiles are written per embedding model and never pruned. Every model a user configures leaves a permanent ~5 MB file in the databases directory, including models they have since removed.This was harmless until #111, because a filename bug meant the cache never actually persisted. Now that it does, the files accumulate for real.
Where
invokeai/app/services/image_index/image_index_default.py::_build_vocab_embeddingswrites:start()already prunes the database side for models that are no longer current —delete_embeddings_for_other_models— but there is no equivalent sweep for these files. Nothing globs the databases directory (the only other consumers,sqlite_migrator_impl.pyandsqlite_database.py, address files by exact name), so the strays are inert; the cost is disk only.Size
Real file from a live install:
cluster_vocab_blake3_613228c1fd904fb8e.npz, 5.1 MB — a(1675, 768) float32matrix plus the fingerprint. So roughly 5 MB per embedding model ever configured, forever.Reproduction
image_index_modelset to a CLIP Vision model; let the vocabulary build. Acluster_vocab_<tagA>.npzappears.image_index_modelto a different model and restart. A secondcluster_vocab_<tagB>.npzappears.Observed:
cluster_vocab_<tagA>.npzremains indefinitely.Expected: it is removed alongside the embeddings that were dropped for the same model.
Suggested direction
Extend the existing
start()cleanup that callsdelete_embeddings_for_other_modelsto also globcluster_vocab_*.npzindb_path.parentand unlink any whose tag is not the current model's. Worth doing as a best-effort pass wrapped in a handler — a failure to delete a stale cache should never prevent startup.Two adjacent details for whoever picks this up: the
[:24]truncation makes the tag lossy (fine for collision purposes — ≥17 hex chars survive — but the sweep must compare truncated tags, not full ids), and any glob should be careful not to delete a.tmp.npzstaging file belonging to a concurrently-running instance. See #121 on staging-file naming.