feat: SDK v2 — typed semantic declarations, generated submissions, and a findings-based curation pipeline

[rorybyrne]

Summary

Redesign the authoring SDK as an executable data standard: archive = osa.Archive() is the composition root; the author writes what a standards document would say — record shapes with semantic field types and declarative constraints, plus a small set of functions for what tables can't express — and the platform generates everything else (storage, indexes, submission pipeline, docs, catalog, MCP). Four decorators total: record, source, curate, derive.

Builds on #180 — its server-side halves (Convention removal, ingesters/ingester_releases provenance symmetry, docs-on-schema, File as field type, one type map) stand unchanged; its §5 SDK surface is superseded by this design. Delivers the metadata-transformation capability #164 frames as tracked silver transforms. Spans two repos (SDK: osa-py; server: this repo), managed from here.

The organizing analogy

FastAPI works because its constructs map one-to-one onto the sections of a document its users were already mentally writing — the API contract. OSA's authors were already writing a document too: the data standard (MIAME, PDB deposition guidelines, a consortium's submission spec, a biotech's CRO-acceptance SOP). The SDK's design test, applied to every construct: which sentence of the standards document does this express? A construct that expresses no such sentence is machinery leaking.

The dimensions an archive author controls are the standard's section headings: identity (what things exist), semantics (what fields mean: types, units, vocabularies), admissibility (what is accepted, and with what reservations), origin (how data enters), derived knowledge (what the archive computes), audience (what consumers can ask — generated, like FastAPI's /docs).

Declarative-first

Real standards are overwhelmingly tables, rarely prose algorithms. The SDK mirrors that: most of a standard compiles to zero functions.

import osa

archive = osa.Archive("pockets")

@archive.record
class Protein(osa.Record):
    """Crystal structures with computed binding pockets."""
    organism: osa.Term[NCBITaxon]                     # vocabulary binding IS the rule:
                                                      #   unresolvable -> rejected, with candidates
    resolution: osa.Quantity["angstrom"] = osa.field(
        range=("0.5 angstrom", "10 angstrom"),
        violation=osa.caveat,                         # severity is a declaration
    )
    isolated_from: str | None = osa.field(missing=osa.caveat)
    structure: osa.File(format="pdb")

    class Docs:                                       # #151 gate unchanged; docstring = purpose
        example_questions = [...]

Field declarations carry constraints (range=, pattern=, missing=) and their severities. Functions exist only where the standard has prose.

The four decorators

@archive.source(Protein, schedule="daily")            # origin
def pdb(cursor: osa.Cursor) -> Iterator[osa.Submission[Protein]]: ...

@archive.curate(Protein.organism)                     # claim-side: fix and judge, one kind
def organism(value: str | osa.Term, taxa: osa.Vocabulary, findings: osa.Findings) -> osa.Term:
    match = taxa.reconcile(value)
    if not match.confident:
        findings.reject("organism/unresolvable", candidates=match.candidates)
        return value
    if match.fuzzy:
        findings.caveat("organism/fuzzy-match", resolved_from=value)
    return match.term

@archive.curate(Protein)                              # record-scoped judgement
def plausible_temperature(s: osa.Submission[Protein], findings: osa.Findings) -> None: ...

@archive.derive(runs="background")                    # fact-side: computed knowledge
def find_pockets(p: Protein) -> list[Pocket]: ...

Decorator	Standard's dimension	Separate because
record	identity + semantics + declarative admissibility	the standard itself
source	origin	different trigger (clock/upstream, not a submission), different JTBD, often a different person
curate	admissibility + semantic resolution — everything done to claims	runs pre-accession; receives Submission[T] / field values
derive	derived knowledge — everything computed from accepted facts	runs post-accession; receives T; records stay immutable while derivations re-run

Division of labor: decorators declare role + config (schedule=, runs=); signatures declare binding (Protein -> list[Pocket] attaches the feature; Protein.organism FieldRef scopes a curate). Binding is strict with loud import-time cross-checks: a @archive.curate function taking Protein (a fact) is an error whose message points at @archive.derive, and vice versa — the claim/fact type split makes "which decorator" mechanically answerable, never a taxonomy judgement.

Curate merges transformation and judgement deliberately (the Pydantic lesson: a validator may fix or complain; users never pick between two kinds). Two channels, both always available: the return value is canonicalisation (field-scoped functions return the possibly-corrected value; provenance records before/after + the producing release); the injected findings collector is commentary at four severities (below). Mixed acts — "fuzzy-matched, accept with caveat" — are first-class. Rejection has exactly one home: a finding.

No fail-fast: all curate functions run; findings accumulate; the submitter gets the complete report in one round trip (decisive when the submitter is an agent).

Findings

Severity	Effect
reject	Refused; findings returned to submitter
hold	Routed to the curation queue; a curator releases or rejects (gives the curation domain its real model)
caveat	Accessioned, but the caveat travels with the record permanently: rendered in catalog/manifest, filterable in queries
note	Accessioned; informational; kept in provenance

Stable machine-readable codes (organism/unresolvable); codes render into generated docs ("submissions are rejected when…") and the #151 docs gate covers them; #203's example-validation applies. Declarative constraint violations produce findings through the same machinery — one severity model end to end.

Submissions: bronze derived from silver

The author never models raw data. Submission[Protein] is generated as the widened shadow of the Record: every field accepts the un-harmonised form (organism: str | Term; resolution: str | Quantity — "2.4 Å" parses), files accept paths. A Submission[T] and a T are different types on purpose: a claim vs a curated fact — and the type split is also the pipeline position, which is what makes the curate/derive cross-check possible.

• Deposition: Protein.submission(...) — structural parse client-side, semantics server-side.
• Ingestion: a source's job is upstream-shape → Submission[Protein]; after that, one pipeline for every entry path. (Server-side registry keeps the feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180 ingesters naming.)
• Raw payloads preserved verbatim forever (bronze = the current records.metadata JSONB, demoted to raw-as-received; typed metadata tables become canonical, written only by the pipeline).

Lifecycle, provenance, semantics types

• Lifecycle: received → curating → { rejected | held | accessioned } → deriving → current, then superseded (.N+1) / retracted (tombstone; access closes, accession + reasons stay citable — DELETE is not in the vocabulary). Re-curation with a new release mints a new record version through the same pipeline; replayable because raw is preserved.
• Provenance chain unchanged (feat: hook versioning, per-row provenance, and M2M deploy #145/feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180): derived row → run → release; record → ingest run → release; now also each curated field value → run → release, with before/after. Staleness queryable: osa recompute --stale.
• Metadata is not a feature table (decision 2026-08-14, supersedes the "metadata-as-feature" framing in feat(ingest): ingester identity + versioned releases (raw-ingestion provenance) #208's scope-correction note): canonical typed metadata keeps its own store family — the existing metadata.<slug>_v<major> tables (own PG schema, keyed per major so additive minors extend in place) — and gains run-provenance columns analogous to features.*.run_id, not residency in the features namespace. This also dissolves feat(ingest): ingester identity + versioned releases (raw-ingestion provenance) #208's PgName identifier-budget blocker, which only arose from prefixing metadata_ into feature-table names.
• Value objects: osa.Quantity (dimension-constrained; any unit of the dimension accepted, canonical unit stored, original in provenance; comparisons dimension-checked), osa.Term (CURIE + label + owning vocabulary; cross-vocabulary comparison is an error), osa.File (content-addressed, lazy handles). Unit-aware filters now; ontology-transitive .within() as roadmap with semantics-domain closure tables. Client query builder compiles to the /data FilterExpr API (not SQL).
• Preflight (design constraint now, feature later): curate functions whose dependencies are pure must be client-runnable — same findings, same codes, before bytes upload. SDK domain layer stays zero-I/O.

Precedents (why this shape)

• FastAPI: constructs map to the contract document's sections; variation lives in signatures; docs generated. Adopted wholesale as the analogy.
• Pydantic: transformation + judgement are one validator kind. Source of the curate merge.
• pandera: parsers (transform) ordered before checks (judge) — validates the transform/judge acts being distinct phases without needing distinct registration kinds.
• Great Expectations: validation phrased as expectation; complete reports, not fail-fast. Source of the findings-accumulation contract.
• sqlmesh audits: pre-promotion gating with blocking/non-blocking modes — the severity gradation, independently invented.
• dbt/Dagster: source as the consensus origin word; Dagster's asset-centric refactor as the data-centric cautionary/corrective tale.
• Rails callbacks: the anti-pattern — naming functions by pipeline position (before_save) instead of domain role.

Design history (rejected shapes, so the reasoning survives)

1. @archive.ingest/transform/derive (feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180 §5): stage names leak execution vocabulary; __name__ binding is stringly.
2. No decorators, pure signature inference: silent-failure registration; nothing greppable.
3. Verb-per-kind (source/harmonise/check/derive): harmonise-vs-derive is a provenance-taxonomy question authors can't answer (JTBD: biotech QC, consortium, FRO, academic — none arrive with that distinction).
4. One @archive.curate for everything: lost the "when does this run" information.
5. Moment names (intake/derive): better, but "intake" is our pipeline's word, not the standard's.
6. Dimension-per-decorator (resolve/expect split): reintroduced two rejection channels and made fix-plus-remark inexpressible → merged back into curate with the findings collector.

Growth rule: a new decorator requires a new position relative to the claim/fact boundary or a new trigger class — not a new use case. Use cases go into the existing four. Integration events (Slack, Nextflow, DOI minting — the outside world subscribing) are a separate seam, trigger-objects-as-data on the existing outbox events, not named lifecycle decorators.

Deliberately not adopted (from the reference design reviewed)

• Collection-scoped registries — declarations attach to the one Archive.
• Derived scalar fields on the Record — one derivation mechanism (Features); serving may project single-row features as record columns (merged in consumption, never in declaration: records are closed, the set of computations open).
• Same-process SQL query builder — OSA is client-server.

Relationship to existing issues

• feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180: server halves proceed unchanged; §5 superseded.
• feat: medallion data architecture — raw landing, tracked silver transforms, declared gold marts #164: curate functions are the tracked silver transforms; verbatim raw = bronze; gold = generated serving surfaces.
• refactor: name feature tables and /data segments after the Feature model, not the producing hook #214: feature naming by model — assumed (list[Pocket] → pocket).
• feat: natural / canonical record identifier (declared entity key + resolve-by-key) #175: natural keys interact with Submission identity — coordinate.
• perf: /data table reads do O(table) work per request — per-request COUNTs, unbounded SQL, unindexable sort #212/perf: stream table dumps via Postgres COPY — snapshot-consistent, resumable, constant-memory #213: independent; land first.
• fix: datetime schema fields can never publish (SDK maps them to date, publish rejects the timestamp) #195 (hard requirement, added 2026-08-17): the semantic type vocabulary / feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180's one type map MUST include a first-class datetime (date with time) type end to end — SDK declaration, storage, projection. Today's silent zero-publish is exactly the SDK-maps-datetime-to-date gap this redesign must not rebuild on.
• fix: missing max_file_size fails as a raw 422 after both images build #200: osa.File(...) declarations carry size constraints (e.g. max_size=) with a sane default, so a missing limit can never surface as a late server-side 422 after image builds.
• fix: osa test doesn't cover the publish stage, so it passes for conventions that publish nothing #202: state the testing story explicitly — findings accumulation makes silent drops structurally impossible (an unstorable/unresolvable value is a reject finding, not a skipped row), and osa test/preflight must exercise the pipeline through accession semantics so fix: datetime schema fields can never publish (SDK maps them to date, publish rejects the timestamp) #195-class bugs are caught client-side.

Open design questions

1. Ordering among curate functions (field-scoped vs record-scoped; declared constraint findings vs function findings) — pandera's parsers-then-checks ordering is a candidate: field canonicalisation → declarative constraints → record-scoped functions.
2. Multi-module archives: how declarations across packages attach to the single Archive (import side-effects vs explicit archive.include(module)) without reintroducing a Collection concept.
3. Quantity storage: canonical-unit numeric column (resolution__angstrom) vs value+unit pair; unit change = conversion migration, not retype.
4. Findings storage/API shape; hold integration with the curation domain.
5. Migration/versioning story for current SDK users (pre-launch: re-ingest per feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180; the API break still needs a version story).
6. Whether @archive.record registration survives or a metaclass auto-registers against the archive (explicitness vs ceremony).

Phasing

1. Value objects + Record declarations + declarative constraints + Submission[T] generation (SDK-only beyond feat: remove Convention — Archive SDK (Schema/Feature/derive/ingest) + ingester provenance #180).
2. Curate functions + findings + lifecycle (server: pipeline stages, findings tables, field-provenance events).
3. Curation queue (hold), retraction, recompute --stale, preflight.