Skip to content

feat: pluggable external metadata sources — Protocol, registry, resolver hook (#784) - #1106

Merged
jepegit merged 1 commit into
masterfrom
784-metadata-sources
Sep 26, 2026
Merged

jepegit merged 1 commit into
masterfrom
784-metadata-sources

Conversation

@jepegit

@jepegit jepegit commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Closes #784 — Epic M / M1 (stage 4 of #783), read path only. Non-yolo: public Protocol + resolver precedence, so this PR is the review point and is not auto-merged.

What

  • cellpy.readers.metadata_sources (new package)
    • MetadataSource Protocol (name: ClassVar[str], fetch(MetaQuery) -> tuple[MetaRecord, ...]), SupportsMetadataPush declared for M3 (unused).
    • MetaQuery, MetaRecord (draft cell / test mappings + external_id / source_uri), ExternalLink (persisted back-link), validate_record (unknown fields, pre-filled provenance and None values are contract violations).
    • cellpy.metadata_sources entry-point registry mirroring the loader registry: lazy, broken plugins skipped with a warning, register() for tests/notebooks, get_source() instantiates once.
    • fetch_meta(source, query, strict=False): null object — unknown/unreachable source or invalid record ⇒ () + warning; MetadataSourceAuthError always propagates.
    • testing.check_metadata_source conformance kit + DictMetadataSource fake.
  • MetaResolver.resolve(external=…) (also resolve_cell_meta / resolve_test_meta / resolve_from_loader_result): external records join the journal/db layer below the journal row (kwargs > journal row > external sources in priority order > raw file > defaults). Resolution.origins, origin_of(), fields_from_origin(); explain() prints mass: journal/db (batbase). Old shape unchanged without externals.
  • CellpyCell.fetch_meta(source, key=None, *, kind="cell_name", project=, apply=True, strict=False, **extra) + external_links property. Applies the first record via the resolver → apply_test_meta_to_legacy (engine's live boxes), records ExternalLink(fields=applied). Data.external_links persisted in v9 meta.json under "external_links"; copied by from_cell.

Decisions (see .issueflows/04-designs-and-guides/metadata-sources.md)

  • Journal row outranks an external source: a user's journal edit is a deliberate local override; the lab DB is authoritative relative to the cycler file, not the user.
  • No new Layer enum member — contributors + origins keep Layer ordering stable.
  • CellMeta.uuid lives in cellpy-core → filed core-first as CellMeta.uuid: cell-level join key for external metadata sources cellpy/cellpy-core#151 rather than changed here.
  • Deferred: batch.from_source, multi-source config, cache/TTL, push, BattINFO vocabulary map.

Tests

  • tests/test_metadata_sources.py: 36 tests, 6 essential (contract shape, null object, auth propagation, precedence + provenance, cell surface).
  • Full suite: 1757 passed, 187 skipped; uv run pytest -m essential: 959 passed.

Docs

HISTORY, docs/api/readers.md, docs/agents/index.md, AGENTS.md agents section, test registry.

Next

cellpy/cellpy-connectors#2 (M2, BatBase MetadataSource adapter over BatBaseClient, registered via the entry point).

Made with Cursor

…ver hook (#784)

- cellpy.readers.metadata_sources: MetadataSource / SupportsMetadataPush
  Protocols, MetaQuery, MetaRecord, ExternalLink, validate_record, error
  hierarchy; cellpy.metadata_sources entry-point registry (register, names,
  get_source); null-object fetch_meta (unknown/unreachable => empty layer,
  auth errors propagate); conformance kit check_metadata_source
- MetaResolver.resolve(external=...): records join the journal/db layer
  below the journal row; Resolution.origins / origin_of / explain() name
  the winning source
- CellpyCell.fetch_meta(source, key=None, kind=, apply=, strict=) and
  external_links; Data.external_links persisted in v9 meta.json
- tests/test_metadata_sources.py (36, 6 essential); docs, HISTORY, design
  note; CellMeta.uuid filed core-first as cellpy/cellpy-core#151

Closes #784

Co-authored-by: Cursor <cursoragent@cursor.com>
@jepegit jepegit mentioned this pull request Sep 26, 2026
10 of 21 tasks
@jepegit
jepegit merged commit 090c719 into master Sep 26, 2026
7 checks passed
@jepegit
jepegit deleted the 784-metadata-sources branch September 26, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pluggable external metadata sources (query an API for cell/test metadata)

1 participant