feat: Function — one abstraction for hooks, ingesters, and transforms (registry, invocations, streaming sessions, candidate adoption)

[rorybyrne]

Summary

Hooks and ingesters (and soon #215's curate-transforms) are three instances of one thing: named, versioned, containerized, user-supplied code whose executions are provenance-tracked. Today we maintain parallel meta-management implementations for each — #217 exists precisely because the two registries diverged. This issue designs the single abstraction — Function — plus the execution-model redesign that reshapes its core entities: long-lived streaming containers replacing per-batch orchestration, and a candidate-enrichment pipeline with atomic adoption replacing the batch publish machinery.

Greenfield: fresh tables, no data migration — re-deploy conventions and re-ingest. Belongs to the #180 PR 3 epoch (identities and edges are already in motion there).

The model

A Function is named, versioned, containerized, user-supplied code. Its contract says what the name is bound to. Its releases are immutable definitions of exactly what can run. Its invocations record exactly what did run. A session is a long-lived unit of work — a release snapshot plus a watermark — spanning 1..N invocations (restarts resume from the watermark).

Function[C]          name: FunctionName (PgName), contract: C, live_release_id
Release              function_name, version, runtime: OciConfig, source_ref, built_by, built_at
Invocation           id (caller-supplied, deterministic on pipeline paths), release_id,
                     status, started_at/finished_at/duration_s, retries, log_ref
Session              id, function/release snapshot, mode, watermark, status,
                     failure_reason/kind — replaces ingest_runs' batch state machine

One FunctionRegistry[C] port + one PostgresFunctionRegistry adapter + one FunctionRegistryService. Storage: functions / function_releases / function_runs / function_sessions, kind enum column, PK (kind, name) (kinds keep separate namespaces), contract as kind-discriminated JSONB. One router factory instantiated per kind (/hooks, /ingesters, later /transforms) with per-kind auth scopes (hooks:write, new ingesters:write); ingesters gain the full registry surface (releases GET/POST, live PUT, runs) they currently lack.

Idempotency (one rule, both kinds): definition-equality against the live release — byte-identical redeploy is a no-op; any difference (image, digest, config, limits, source_ref; built_by excluded) mints vN+1. No (name, digest) unique constraint (a config-only redeploy legitimately reuses a digest). This is #208's ingester semantics; #217 converges hooks before this abstraction freezes them.

The two (three) instantiations

Kind	Contract type	Contract equality	Conflict	Name policy
hook	FeatureContract(TableFeatureSpec)	model equality	hook_contract_conflict	reserved-name check (URL slot — migrates to feature names under #214)
ingester	SchemaBinding(LocalId)	id equality	ingester_schema_conflict	none
transform (#215)	FieldContract (schema field set)	model equality	tbd	none

The contract carries equality, the typed conflict error (message + code), and name policy. Extension rule: a new Function kind = a new contract type (+ execution manifest + provenance-FK target). Never a new registry.

Divergence audit (input evidence — how every current difference resolves)

From a method-by-method diff of the two registries (28 divergences), classified:

• Accidental — erased by one implementation (8): assert vs InfrastructureError after inserts (asserts vanish under -O; raise wins), missing error code=s, HookName(root='x') leaking into messages, join-direction, verbatim-copied Outcome type, untyped get_release_by_id.
• Parametric — the seams (contract, conflict semantics, name policy, route prefix, auth scope): typed parameters, no booleans.
• Decisions — one behavior, not a strategy knob:
  • Idempotency: definition-equality (above); hook digest-only rule is bug fix: hook release idempotency ignores config changes — config-only redeploys silently never take effect #217, not a policy.
  • resolve_live: batch-with-omission (list[name] → dict) + single require_live(name) raising NotFoundError — deletes today's duplicated call-site raises (validation.py:100, process_batch.py:415); PermanentError wrapping stays at the workflow call site.
  • Invocation ids: caller-supplied; pipeline paths derive deterministic ids (uuid5 over {session}:{chunk}:{name}) so record_invocation's ON CONFLICT DO NOTHING idempotency and crash-recovery guards are real. The deposition path's uuid4 (validation.py:110) is inconsistent with this and gets fixed here.
  • OOM escalation: HookRelease.with_doubled_memory relocates to the runner layer (escalation = a runner deriving a bumped OciConfig; refactor: untangle hook/ingestion runtime failure handling into facts → policy → action #152's policy already owns the budget). Invocation.retries generic.
• Exclusion: ingest_runs' batch state machine is orchestration, not a Function concept — and it is deleted outright by the execution redesign below, replaced by the generic Session.

Execution redesign: streaming sessions, no batches

Two execution contracts (the "modes"):

• oneshot — mount inputs, run to completion, collect outputs. Deposition path today, unchanged.
• streaming — a long-lived container runs until its work domain is exhausted, continuously emitting chunks + a watermark to the output dir (FS contract preserved: chunk files + watermark file; no network). Ingester: streams candidates from upstream until the cursor dries up. Enrichment functions: stream over the candidate flow. Recompute/backfill: stream over published records.

Supervision replaces choreography: liveness = watermark advance (no progress in N min → policy), crash/eviction mints a new Invocation resuming from the session's committed watermark (provenance gets more precise — post-restart rows stamp the restart's invocation), and failure handling is the existing #152 machinery at container granularity (Retry / RetryWithMoreMemory / GiveUp / AbortSession) with restart budgets.

Where each of the batch's jobs goes:

Batches today provide	New home
Orchestration unit (events, counters, redelivery)	Deleted — this was the accidental job
Crash-redo blast radius	Session watermark; spool + idempotent publish converge
Publish transaction sizing	The chunk — demoted to a data-plane detail inside the publisher loop
Progress observability	Watermark + session status (what #204 wants to show anyway)
Container timeout bounds	Progress-based liveness + restart budgets (see risks)

The pipeline: candidates, enrichment, atomic adoption

Records are immutable, and validation decisions may depend on derived features — so mint-then-reject must be unrepresentable, and everything a verdict can read must exist before the mint. (This is also current behavior: hooks gate publication today.) The candidate is the entity that lives through the pipeline, accumulating enrichment; the record is its final frozen form:

candidate → harmonise → derive (features computed ON the candidate)
          → check (findings — may consume computed features, declared via signature DI:
                   def plausible(s: Submission[Protein], pockets: list[Pocket]) -> Findings)
          → verdict
              ├─ rejected:    candidate + findings + computed features retained in bronze; no record
              ├─ held:        fully-enriched candidate parks in the curation queue
              └─ accessioned: ONE transaction mints the record, adopts its features,
                              updates table_statistics, advances the watermark

• The curate/derive distinction is what a function touches, not when it runs: curate fixes/judges asserted content; derive computes attached data and never touches asserted fields. Both run pre-mint; derive additionally re-runs post-mint for recompute under a new release (no record version bump). The same function serves both positions — it is pure over the harmonised shape (Protein) and is not told whether accession happened.
• Enrichment is a dependency graph ordered by signatures (harmonisers → derivations → checks that consume them).
• The publisher is the join point: candidate's enrichment set complete → verdict → atomic adoption per chunk. Records are born with their features; fix: partial publish failure leaves records unreadable and unrepairable #196 (partial publish) and fix: ingest run that published nothing reports COMPLETE with a non-zero count #197 (lying counts) become structurally impossible.
• Amends feat: SDK v2 — typed semantic declarations, generated submissions, and a findings-based curation pipeline #215: derive semantics (runs on candidates pre-mint + records post-mint), findings-consume-features DI, lifecycle order (received → harmonising → deriving → checking → verdict).

Simplification ledger (what gets deleted)

• NextBatchRequested → ProcessBatch choreography (~700 lines: stage guards, redelivery semantics, mark_batch_ingested GREATEST logic, batch counters, ingestion_finished latch)
• ingest_runs state machine → generic function_sessions
• IngestBatchPublished as a stats carrier (publisher updates table_statistics lockstep in its own transaction — perf: /data table reads do O(table) work per request — per-request COUNTs, unbounded SQL, unindexable sort #212 D2 simplifies)
• Duplicated deterministic-run-id helpers; duplicated "no live release" raises
• The entire second (and future third) registry implementation; fix: hook release idempotency ignores config changes — config-only redeploys silently never take effect #217's bug class becomes unwritable
• Hook lockstep: enrichment functions proceed per-candidate, not per-batch-all-hooks

Risks / new design obligations

1. Hung-container detection: watermark-based liveness + restart budgets (crash-loop gives up via policy).
2. Eviction exposure of multi-hour pods: feat: annotate hook/ingester Job pods safe-to-evict=false so node consolidation never kills a long run #211 (safe-to-evict=false) becomes a requirement of this design, absorbed here.
3. In-flight loss on restart = work since last committed chunk — bounded by chunk size (tunable).
4. Publisher join semantics: per-candidate completeness tracking across N enrichment sessions.
5. Operator-initiated cancellation (feat: cancel running ingest runs #123): AbortSession exists above only as a failure-policy verb — the design must also expose a user/API-facing cancel on sessions (status transition, watermark semantics for cancelled work, cleanup of in-flight invocations).

Open questions

1. Re-derivation vs re-judgement: a new release recomputes features on published records; new values might fail a check that passed at admission. Records immutable; retraction exists. Proposal to evaluate: recompute never silently un-publishes — it may attach caveat findings to provenance; judgement is an admission-time event. The one real tension between "checks consume features" and "features are re-derivable".
2. Held candidates park fully enriched — candidate storage holds feature data pre-adoption (the spool, formalized). Storage shape for bronze candidates + findings + pre-adoption features.
3. Session scheduling (feat: cron-scheduled ingestion runs with declared update strategy (additive | overwrite) #154's cron + update strategy) as session-level config on the identity.
4. Function module home (shared kernel vs new domain) and whether the deposition oneshot path shares the publisher.

Relationship to existing issues

• fix: hook release idempotency ignores config changes — config-only redeploys silently never take effect #217: interim mirror fix lands in feat(ingest): ingester identity + versioned releases (raw-ingestion provenance) #208 so both registries are correct before being unified; this issue then deletes both copies.
• feat: SDK v2 — typed semantic declarations, generated submissions, and a findings-based curation pipeline #215: amended by the pipeline section (derive timing, findings-consume-features, lifecycle).
• feat(ingest): ingester identity + versioned releases (raw-ingestion provenance) #208: sessions pin their release at start; invocations inherit it — structurally closes the "snapshot does not pin execution" gap (Greptile P1 on feat(ingest): ingester identity + versioned releases (raw-ingestion provenance) #208) without patching ProcessBatch.
• feat: annotate hook/ingester Job pods safe-to-evict=false so node consolidation never kills a long run #211: absorbed as a requirement. feat: osa ingestion status / list — observe a run after starting it #204: reads the simpler session model. feat: cron-scheduled ingestion runs with declared update strategy (additive | overwrite) #154: session scheduling. fix: partial publish failure leaves records unreadable and unrepairable #196/fix: ingest run that published nothing reports COMPLETE with a non-zero count #197: structurally mooted (repair half of fix: partial publish failure leaves records unreadable and unrepairable #196 unaffected). feat: medallion data architecture — raw landing, tracked silver transforms, declared gold marts #164: the pipeline is its medallion made executable. feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180 PR 3: same epoch; conventions' hook/ingester edges land on Function identities.

Addendum 2026-08-17 — issues this design also resolves or reshapes (from the full-tracker triage):

• feat: cancel running ingest runs #123: cancellation — absorbed as design obligation 5 above; close against this when sessions land.
• feat: adaptive backpressure tuning based on empirical hook execution times #122: obsolete as written — the batch machinery it tunes is in the simplification ledger. If adaptive pacing is still wanted, re-frame it against streaming sessions (chunk sizing, liveness thresholds).
• fix: make ingest batch accounting idempotent (complete_batch/fail_batch) #148 / fix: potential duplicate IngestCompleted under concurrent batch completion #124: mooted — the batch counters, mark_batch_ingested GREATEST logic, and ingestion_finished latch they patch are all deleted; close both when this lands.
• refactor: converge deposition + ingestion hook-execution paths onto one provenance mapping #150: delivered — the single registry + unified Invocation recording removes the two hand-mirrored provenance mappings; the divergence audit above already names its validation.py:110 citation.
• refactor: deposition validation should retry transient infra failures via the failure policy #155: prerequisite (deterministic run identity) delivered by caller-supplied deterministic invocation ids; the remaining retry-policy ask folds into open question 4 (whether the deposition oneshot path shares the publisher/failure handling).
• refactor: saga-based validation for per-hook commits and observable progress #67: substantively delivered — hook lockstep is deleted (per-candidate enrichment), and progress observability = watermark + session status.