feat: pluggable external metadata sources — Protocol, registry, resolver hook (#784)

.issueflows/03-solved-issues/issue784_original.md

@@ -0,0 +1,34 @@
+# Issue #784: Pluggable external metadata sources (query an API for cell/test metadata)
+
+Source: https://github.com/jepegit/cellpy/issues/784
+Labels: enhancement, v2, cellpy2-stage5
+Captured: 2026-09-26 (Epic M / M1, stage 4 of #783; read scope only)
+
+---
+
+Let cellpy **pull (and optionally push) cell/test metadata from an external source** — starting
+with **BatBase** (our in-house Django + PostgreSQL app, HTTP API) — but **source-agnostic**: a
+pluggable contract so other labs' stores work too. First transport assumption: an **HTTP API**
+(GET to read, POST/PUT to write).
+
+Realizes the deferred **metadata-plan Step 7** ("DB/API transport — keep the door open"). Full
+design: **[cellpy2-metadata-source-integration.md](https://github.com/cellpy/architecture-plan/blob/main/cellpy2-metadata-source-integration.md)**.
+
+**Approach (mirrors the loader contract):**
+- `MetadataSource` **Protocol** + `cellpy.metadata_sources` entry-point registry — third
+  parties plug in without importing a cellpy base class.
+- Feeds the existing **`MetaResolver` JOURNAL/DB layer** (authoritative lab metadata already
+  outranks raw-file values; provenance extended to name the source).
+- **Anti-corruption boundary**: cellpy pins the Protocol + `MetaRecord`; the **BatBase adapter
+  absorbs API churn** (BatBase API isn't settled yet — adapter can even be a separate plugin
+  package, keeping the HTTP dep out of cellpy core).
+- Adds **`CellMeta.uuid`** + per-source `external_id`/`source_uri` back-link (the identity
+  parking item, metadata-plan OQ6); vocabulary normalization toward **BattINFO** (OQ4).
+- Null-object: an unreachable/unknown source ⇒ empty layer, cell still loads. Push is
+  side-effecting → **explicit opt-in only**, never automatic. Auth via env-only `SecretStr`.
+
+**Scope:** read path first (Protocol + `fetch` + resolver + BatBase GET adapter); push a later
+opt-in follow-on. **Post-2.2 (2.3+)** — additive, gates nothing in 2.2.
+
+Adjacent: #243 (dbreader refactor). See design doc for the work breakdown + open questions
+(query key, adapter placement, cache/offline, push permission, multi-source precedence).

.issueflows/03-solved-issues/issue784_plan.md

@@ -0,0 +1,33 @@
+# Plan: #784 MetadataSource Protocol + resolver hook (M1, read path)
+
+Non-yolo (public Protocol + resolver precedence). Executed under the
+maintainer's "start" after M0 shipped; the PR is the review point and is
+**not** auto-merged.
+
+## Scope (from epic783 M1 spec)
+
+`MetadataSource` Protocol, `cellpy.metadata_sources` entry-point registry,
+`MetaResolver` journal/db-layer hook with provenance naming the source,
+null-object on failure, back-link (`external_id` / `source_uri`). No push.
+
+## Approach
+
+1. `cellpy/readers/metadata_sources/` — `contract.py` (Protocols, `MetaQuery`,
+   `MetaRecord`, `ExternalLink`, errors, `validate_record`), `registry.py`
+   (entry points, `register`, `get_source`, `fetch_meta`), `testing.py`
+   (`check_metadata_source`, `DictMetadataSource`).
+2. `meta_resolver.py`: `external=` on `resolve*`; `Resolution.origins`.
+3. `CellpyCell.fetch_meta` + `external_links`; `Data.external_links`;
+   v9 `meta.json` `"external_links"` round-trip; clone copy.
+4. Tests `tests/test_metadata_sources.py` (essential markers on the
+   precedence / null-object / cell-surface checks).
+5. Docs: HISTORY, `docs/api/readers.md`, `docs/agents/index.md`, `AGENTS.md`
+   agents section, design note `metadata-sources.md`, test registry.
+
+## Deviations from the issue text
+
+- `CellMeta.uuid` is a cellpy-core model field → filed as a core-first issue
+  instead of a cellpy change (see status). Back-link lands on the cellpy
+  side (`ExternalLink`) and is persisted.
+- `batch.from_source`, vocabulary normalisation, cache/TTL: deferred (design
+  §4 items 4–5; M3+).

.issueflows/03-solved-issues/issue784_status.md

@@ -0,0 +1,31 @@
+# Issue #784 status (M1 — read path)
+
+- [x] Done
+
+## What's done
+
+- `cellpy/readers/metadata_sources/` package: `MetadataSource` /
+  `SupportsMetadataPush` Protocols, `MetaQuery`, `MetaRecord`, `ExternalLink`,
+  `validate_record`, error hierarchy (`MetadataSourceError`,
+  `MetadataSourceAuthError`, `UnknownMetadataSource`); entry-point registry
+  (`cellpy.metadata_sources`, `register`, `names`, `get_source`); null-object
+  `fetch_meta`; conformance kit `testing.check_metadata_source` +
+  `DictMetadataSource`.
+- `MetaResolver.resolve(external=…)` (and `resolve_cell_meta` /
+  `resolve_test_meta` / `resolve_from_loader_result`): external records join
+  the journal/db layer below the journal row; `Resolution.origins`,
+  `origin_of`, `fields_from_origin`, `explain()` name the source.
+- `CellpyCell.fetch_meta(...)` + `external_links`; `Data.external_links`;
+  v9 `meta.json` `"external_links"` round-trip; clone copy in `from_cell`.
+- Tests: `tests/test_metadata_sources.py` (36; 6 essential). Full suite
+  1757 passed / 187 skipped; `-m essential` 959 passed.
+- Docs: HISTORY, `docs/api/readers.md`, `docs/agents/index.md`, `AGENTS.md`
+  agents section, `metadata-sources.md` design note, test registry.
+- `CellMeta.uuid` filed core-first as cellpy/cellpy-core#151 (model lives in
+  cellpy-core; needs release + re-pin).
+
+## Remaining work
+
+None on the cellpy side for the read path. Review the PR (non-yolo). Follow-ups:
+cellpy/cellpy-connectors#2 (BatBase adapter, M2), cellpy/cellpy-core#151,
+M3 push (2.3), `batch.from_source` / multi-source config (design §4 item 5).

.issueflows/04-designs-and-guides/metadata-sources.md

@@ -0,0 +1,56 @@
+# External metadata sources (Epic M, read path)
+
+Issue: #784 (M1; stage 4 of #783). Design origin:
+`cellpy-design-and-development/active/cellpy2-metadata-source-integration.md`.
+Adapter side: `cellpy-connectors` (#1 `BatBaseClient`, #2 adapter).
+
+## Context
+
+cellpy 2 reserved the seam — `MetaResolver` (`kwargs > journal/db > raw file >
+config defaults`) with per-field `Resolution` provenance — but nothing outside
+the journal fed it, and the resolver is not yet on the raw-load hot path
+(`cellreader.from_raw` still fills the legacy meta boxes directly). A lab's
+system of record (BatBase first) should contribute mass, area, nominal
+capacity, project without cellpy depending on one lab's API.
+
+## Decisions
+
+| Topic | Decision |
+| --- | --- |
+| Package | `cellpy/readers/metadata_sources/` — `contract.py`, `registry.py`, `testing.py`; re-exported from the package |
+| Contract | `MetadataSource` Protocol: `name: ClassVar[str]`, `fetch(MetaQuery) -> tuple[MetaRecord, ...]`. Structural, `runtime_checkable`; no base class. `SupportsMetadataPush.register(record) -> str` declared, unused (M3). |
+| Shapes | `MetaQuery(key, kind="cell_name", project, extra)`; `MetaRecord(source_name, external_id, source_uri, cell={CellMeta field: value}, test={TestMeta field: value}, fetched_at, raw)`; `ExternalLink` = persisted back-link |
+| Validation | `validate_record`: unknown field names, pre-filled provenance (`uuid`, `source_*`, …) and `None` values are errors — a typo in an adapter's field map fails its conformance test rather than losing a mass |
+| Registry | `cellpy.metadata_sources` entry-point group; lazy discovery; entry may be a class (instantiated once, no args) or a zero-arg factory; broken plugins skipped with a warning; `register()` for tests/notebooks and pre-configured instances |
+| Null object | `fetch_meta(source, query, strict=False)`: unknown/unreachable source or invalid record ⇒ `()` + warning. **`MetadataSourceAuthError` always propagates** — a refused token is the user's to fix, hiding it would look like "no data". `strict=True` raises everything. |
+| Resolver | `MetaResolver.resolve(external=…)`: records join the **journal/db layer below the journal row** (kwargs > journal row > external sources (priority order) > raw file > defaults). `Resolution.origins[field]` names the contributor (`"batbase"`, `"journal"`); `origin_of()`, `fields_from_origin()`, `explain()` show it. Old provenance shape unchanged when no externals are given. |
+| Cell surface | `CellpyCell.fetch_meta(source, key=None, *, kind, project, apply=True, strict=False, **extra)`; `key` defaults to `cell_name`. Applies the **first** record (warns if several) through `resolve_*_meta(external=record)` → `test_meta.apply_test_meta_to_legacy` so the engine's live legacy boxes are updated; `external_links[source] = ExternalLink(fields=applied)`. Returns all records. |
+| Persistence | `Data.external_links: dict[str, ExternalLink]`; v9 `meta.json` key `"external_links"` (omitted when empty); copied by `CellpyCell.from_cell` clone; `apply_meta_document` restores. `raw` payloads are never persisted. |
+| Not done here | `CellMeta.uuid` (lives in cellpy-core → core-first issue), `batch.from_source`, multi-source priority config, cache/TTL, push, BattINFO vocabulary map |
+
+## Precedence rationale
+
+Journal row above external source: a value the user corrected in their
+journal is a deliberate local override; the lab database is authoritative
+*relative to the cycler file*, not relative to the user. Both beat the raw
+file. Revisit when multi-source config lands (design §6).
+
+## Alternatives rejected
+
+- A fifth `Layer.EXTERNAL` between RAW_FILE and JOURNAL — the design says the
+  source is a *contributor to the journal/db layer*; a separate layer would
+  change `Layer` ordering that existing provenance tests and docs rely on.
+  Contributors + `origins` keep the enum stable and still answer "who won".
+- Swallowing auth errors like other failures — violates "auth failure
+  surfaces a clear error, not a silent empty" (design §3.6).
+- `fetch(key) -> MetaRecord | None` (epic wording) — the design's tuple form
+  matches the loader contract and lets a tag match several tests.
+- Applying all returned records — ambiguous merge order; first-record +
+  warning keeps it explicit, adapters narrow the query instead.
+
+## Conformance kit
+
+`metadata_sources.testing.check_metadata_source(source, known=, unknown=)`:
+capabilities, unknown key ⇒ `()` (never raises), records validate and carry
+`source_name == source.name`, determinism. `DictMetadataSource` is the
+in-process fake for adapter tests.

.issueflows/04-designs-and-guides/test-registry.md

@@ -256,6 +256,14 @@ current issue**. `/iflow-doctor` may audit the whole suite against this table.
 | tests/test_cli_connectors_plugin.py::test_help_lists_installed_connectors_without_importing_it | yes | yes | live cellpy-connectors stub | #1060 | skip if dist missing (conda) |
 | tests/test_cli_connectors_plugin.py::test_connectors_ping_runs_the_installed_plugin | yes | yes | cellpy connectors ping | #1060 | skip if dist missing (conda) |
 | tests/test_cli_connectors_plugin.py::test_import_cellpy_does_not_import_connectors | yes | yes | import cellpy isolation | #1060 | skip if dist missing (conda) |
+| tests/test_metadata_sources.py::test_protocol_is_structural | yes | yes | metadata_sources.contract Protocols | #784 | public contract shape |
+| tests/test_metadata_sources.py::test_fetch_meta_returns_validated_records_with_timestamp | yes | | metadata_sources.registry.fetch_meta | #784 | |
+| tests/test_metadata_sources.py::test_fetch_meta_unknown_source_is_empty_layer_unless_strict | yes | yes | fetch_meta null object | #784 | cellpy law: empty layer, cell still loads |
+| tests/test_metadata_sources.py::test_fetch_meta_auth_error_is_never_swallowed | yes | yes | fetch_meta | #784 | security posture |
+| tests/test_metadata_sources.py::test_external_beats_raw_file_and_defaults_but_not_journal_or_kwargs | yes | yes | MetaResolver external= precedence | #784 | precedence + origins provenance |
+| tests/test_metadata_sources.py::test_cell_fetch_meta_applies_record_and_links | yes | | CellpyCell.fetch_meta / external_links | #784 | uses `cell` fixture |
+| tests/test_metadata_sources.py::test_external_links_survive_save_and_load | no | | v9 meta.json external_links | #784 | save/get round-trip |
+| tests/test_metadata_sources.py (other 29) | no | | contract validation, registry discovery, conformance kit, resolver ordering | #784 | offline |
 
 **Columns**
 

AGENTS.md

@@ -379,6 +379,12 @@ Quick facts:
   the raw file did not change on disk. Also works after `cellpy.get(".cellpy")`.
   Follow on an interval: `cellpy.utils.live.poll(c, interval=60, on_update=cb)`;
   batch: `b.refresh()` / `b.poll(interval=, on_update=)`.
+- External metadata (lab DB / BatBase): `c.fetch_meta("batbase", key=None,
+  kind="cell_name")` pulls mass / area / nominal capacity / project onto the
+  cell like a journal row and records `c.external_links["batbase"]`;
+  unreachable source ⇒ `()` and no change (`strict=True` raises). Sources:
+  `cellpy.readers.metadata_sources.names()`; the BatBase adapter is in
+  `cellpy-connectors`.
 - Frames: `c.data.raw` / `.steps` / `.summary`; columns via `c.schema.*`.
   After a raw load, each cycle's raw capacity starts at 0. A forgotten tester
   reset that 1.x plotted as doubled capacity is rebased on load for every

HISTORY.md

@@ -2,6 +2,21 @@
 
 ## [Unreleased]
 
+* Pluggable external metadata sources (read path). New
+  `cellpy.readers.metadata_sources`: a `MetadataSource` Protocol
+  (`name`, `fetch(MetaQuery) -> tuple[MetaRecord, ...]`), the
+  `cellpy.metadata_sources` entry-point registry (`register`, `names`,
+  `get_source`), and a null-object `fetch_meta` (unreachable or unknown
+  source ⇒ empty layer; auth errors still raise). `MetaResolver` takes
+  `external=` records into the journal/db layer *below* the journal row and
+  `Resolution.origin_of()` / `explain()` name the source that won a field.
+  `CellpyCell.fetch_meta(source, key=None, kind=, apply=, strict=)` pulls a
+  record onto the cell and records an `ExternalLink` in
+  `c.external_links`, persisted in v9 `meta.json` (`"external_links"`).
+  `metadata_sources.testing.check_metadata_source` is the conformance kit
+  for adapters. No push. The BatBase adapter ships in `cellpy-connectors`.
+  (#784)
+
 * Optional `SupportsIncrementalLoad` protocol (`load_since`) with
   `LoadMarker` and `IncrementalChunk`, hosted in cellpy. Shipped loaders
   stay full-read until they opt in. (#779)

cellpy/readers/cellpy_file/meta_archive.py

@@ -93,6 +93,12 @@ def build_meta_document(
     }
     if frames_had_test_id is not None:
         doc["frames_had_test_id"] = dict(frames_had_test_id)
+    links = getattr(data, "external_links", None) or {}
+    if links:
+        doc["external_links"] = {
+            name: (link.to_dict() if hasattr(link, "to_dict") else dict(link))
+            for name, link in links.items()
+        }
     return doc
 
 
@@ -168,6 +174,16 @@ def apply_meta_document(data: "Data", meta_doc: Mapping[str, Any]) -> None:
     active_id = int(meta_doc.get("active_test_id", 0))
     data._active_test_id = active_id
 
+    links_doc = meta_doc.get("external_links") or {}
+    if isinstance(links_doc, Mapping):
+        from cellpy.readers.metadata_sources.contract import ExternalLink
+
+        data.external_links = {
+            str(name): ExternalLink.from_dict(payload)
+            for name, payload in links_doc.items()
+            if isinstance(payload, Mapping)
+        }
+
     tests_doc = meta_doc.get("tests") or {}
     cell_fallback = meta_doc.get("cell") or {}
 

cellpy/readers/cellpy_file/v9.py

@@ -169,6 +169,7 @@ def save(
     scratch._extra_tests = dict(getattr(data, "_extra_tests", {}) or {})
     scratch._active_test_id = data.active_test_id
     scratch._provenance = dict(getattr(data, "_provenance", {}) or {})
+    scratch.external_links = dict(getattr(data, "external_links", {}) or {})
     scratch.raw_units = dict(data.raw_units)
     scratch.raw_limits = dict(data.raw_limits)
     scratch.raw_data_files = list(data.raw_data_files or [])