feat: vector features and similarity search — pinned embedder, near operator, serving sidecars

[rorybyrne]

Summary

Land vector features and similarity search as an extension of the existing
/data/ read surface rather than as a second query plane: an embedding is a
vector feature column that declares the embedder it lives in, and search is
a near term in FilterExpr that the server resolves using that pinned
embedder. Write/read vector-space parity becomes structural — there is only ever
one declaration — instead of a convention two code paths have to honour.

Split out of #180, which is now scoped to the SDK's Schema/Feature/transform taxonomy
model. This issue owns everything read-time: vector storage, query-time
encoding, the near operator, and the long-lived serving sidecars that
tenant-authored encoders will need.

Why not a Query primitive

The original #180 draft proposed @archive.query — author-declared, named,
parameterised read-time queries with a builder DSL, published as MCP tools and
OpenAPI ops, backed by a serving runtime. Reasons to reject that shape:

1. It duplicates a surface that already exists. FilterExpr
   (server/osa/domain/data/model/filter.py) is a recursive discriminated union
   with typed field refs, full And/Or/Not, and nine operators. Combined
   with feat: server-side cross-feature predicates on the /data records stream #169 (cross-feature predicates) and feat: serve pushdown aggregations (count/min/max/mean/group-by) over the data surface #177 (pushdown aggregations),
   everything in the motivating example except the embedding step is already
   expressible.
2. Auto-publication is free if you extend the grammar, and expensive if you
   don't. FilterExpr has three consumers that import it directly — the REST
   POST body (routes/data/_params.py:12-27), the MCP tool arg models
   (application/api/mcp/models.py:16-17), and the filter-panel facets. A new
   term propagates into the JSON Schema shown to models automatically. A
   separate query plane would need all three wired by hand.
3. Named queries make the author guess the consumer's questions. The whole
   premise of the agent-first read surface is that the caller composes. Teaching
   the archive its canonical questions is a documentation job, and feat: serve an auto-generated agent skill (SKILL.md) from each node #151's
   worked Example blocks already do it (see fix: feature-table naming documented wrong, and Example queries are never validated #203 for making them validated
   rather than opaque).

What is genuinely missing is not named queries. It is query-time compute:
turning a runtime argument into something the planner can use. For similarity
search that is exactly one thing — text (or a molecule, or a sequence) to a
vector.

Design

1. The embedder is pinned to the column, not to a query

A vector feature column declares which embedder it lives in. The embedder
reference (name + digest + dimensionality) is part of the column's type, so
"which space is this vector in" is checked rather than promised, and the read
path resolves the same embedder the write path used by construction.

Storage is an ordinary pgvector column in the features schema with an HNSW
index — no separate vector store (the ChromaDB consistency problem removed in
#137; tests/unit/test_no_index_deps.py currently enforces its absence and will
need revisiting). Provenance is the existing run_id → hook_run → hook_release
chain (#145), same as every other feature row.

Prior art on file: docs/ideas/query-api-sdk-extensions.md proposes exactly this
column-annotation shape (Field(search="vector", distance="cosine"), Vector[N]);
docs/ideas/embedding-service.md proposes the shared POST /embed service.

2. near as a read-surface term

The read request gains a similarity term naming a vector column and a raw input:

{"near": {"field": "features.strain_embedding.vector", "to": "thermophilic archaeon"},
 "filter": {"kind": "predicate", "field": "features.assembly.checkm_completeness",
            "op": "gte", "value": 90},
 "limit": 10}

The server encodes to with the embedder pinned on that column and compiles to
ORDER BY vec <=> $q LIMIT k. A literal vector is accepted in place of text for
callers that already have one.

Note this is an ordering + bound, not a boolean predicate, so it sits
alongside filter rather than inside And/Or.

3. Prerequisites in the read store

Two real constraints, both worth naming before anyone estimates this:

• feat: server-side cross-feature predicates on the /data records stream #169 is a hard dependency. A near term on a feature column combined with
  a metadata predicate is a join, and there is no join from metadata.* to
  features.* today — the grammar admits it and the compiler rejects it in
  three places (postgres_table_read_store.py:359-366, :376-381, :421-427).
  Doing feat: server-side cross-feature predicates on the /data records stream #169 forces the two parallel compilers (_compile_filter and
  _compile_feature_filter) to merge, which is overdue regardless.
• A bounded top-k path has to be built. The read store deliberately never
  emits LIMIT; bounding happens by consumption through a server-side cursor
  (postgres_table_read_store.py:142-153), which is what makes multi-GB
  .csv.gz dumps work from the same code. Top-k similarity is a different shape
  and needs its own path — without breaking the streaming one.

4. Embedder resolution, in two stages

Stage 1 — platform-owned text embedder. One optional sidecar with a fixed
POST /embed contract. The server calls it on publish (to populate the vector
column) and on query (to encode near.to). One model, one call path, parity by
construction, and no need to punch a network hole in hooks — which currently run
with dns_config.nameservers = ["127.0.0.1"] (infrastructure/k8s/runner.py:352).

With no embed service configured the feature degrades cleanly: vector columns
still store, near still accepts a literal vector, and text input returns a
clear 422 rather than a wrong answer.

Stage 2 — tenant-owned encoders. Domain-specific encoding (protein structure,
molecule fingerprints, images) can't be a platform capability; the tenant has to
supply the model, and it has to run on both the write and read path. This is
what needs long-lived serving containers.

5. Serving-mode runner (stage 2)

The hook runner is strictly batch-Job-shaped today: build a V1Job, wait for
completion, read output/{features,rejections,errors}.jsonl, delete the Job
(infrastructure/k8s/runner.py; infrastructure/oci/runner.py).
HookRelease.runtime is an OciConfig with no ports or health-check concept
(domain/validation/model/hook_release.py:30-42). Nothing resembles a warm
service.

Adding a second execution mode: stand up a long-lived Deployment + Service (k8s)
or docker run -d (local), with readiness/health, discovery, and a
request/response protocol over the same declared contract. Queries hit the warm
service — never a fresh Job per query. Keep-warm versus scale-to-zero is a config
knob.

This is the largest single item here and should not gate stages 1–4.

6. Publication

Extending the grammar publishes to MCP and OpenAPI automatically (see "Why not a
Query primitive" above). SKILL.md does not follow automatically — its
"Access" block is a hardcoded string (domain/data/service/skill_renderer.py:136-153),
not derived from the route table or format registry. It needs to become
data-driven or no agent will discover similarity search exists.

Phasing

1. feat: server-side cross-feature predicates on the /data records stream #169 — cross-feature predicates; merge the two filter compilers. (Existing issue, prerequisite.)
2. Vector column + pinned embedder — pgvector storage, HNSW, embedder reference on the column, deploy-time validation. Requires the SDK-side Vector type from feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180.
3. near term + bounded top-k path — grammar, planner, read store; literal-vector input only.
4. Platform embed sidecar — POST /embed, called on publish and on query; text input to near starts working.
5. SKILL.md access block made data-driven.
6. Serving-mode runner + tenant-owned encoders — long-lived Deployment/Service lifecycle, read-time invocation, keep-warm policy.

Open questions

• Does the vector column live in the normal features.* table alongside scalar columns, or in its own table? (Wide rows with a 1536-dim vector affect scan cost for non-vector queries.)
• Distance metric — declared per column (cosine/l2/ip) or fixed to cosine for v1?
• Re-embedding on model change: the pinned digest makes the mismatch detectable; what's the repair path? Presumably re-running the transform, which is the reconciliation loop the Living Conventions design already calls for.
• Caching for near: keyed on (column, input-hash), with what TTL? Encoding the same query string repeatedly is the common agent pattern.
• Does near compose with feat: serve pushdown aggregations (count/min/max/mean/group-by) over the data surface #177 aggregations, or is top-k terminal?
• Full-text search (tsvector + GIN) is the same shape of problem — one column annotation, one read-surface term. Land it here or separately?

Related

• feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180 SDK Schema/Feature/transform model (Convention removal) (this issue was split out of it; supplies the Vector type and the embedder declaration)
• feat: vector features, Query abstraction, and domain-specific similarity search #70 vector features, Query abstraction, domain-specific similarity search (superseded in shape by this issue)
• feat: expressive query surface for LLM and agent consumers #125 expressive query surface for LLM/agent consumers
• feat: server-side cross-feature predicates on the /data records stream #169 cross-feature predicates — prerequisite
• feat: serve pushdown aggregations (count/min/max/mean/group-by) over the data surface #177 pushdown aggregations
• feat: hook versioning, per-row provenance, and M2M deploy #145 hook versioning + per-row provenance
• feat: Kubernetes runner and S3 storage adapters #92 k8s runner · feat: unified /data/ read surface (delete index/search/export, rename discovery → data) #137 unified /data/ surface (removed the previous vector stack)
• docs/ideas/query-api-sdk-extensions.md, docs/ideas/embedding-service.md