Skip to content

feat: file pointers from external metadata sources — cellpy.get(source=) and batch.from_source (#1107) - #1119

Merged
jepegit merged 3 commits into
masterfrom
1107-file-pointers-from-source
Oct 1, 2026
Merged

jepegit merged 3 commits into
masterfrom
1107-file-pointers-from-source

Conversation

@jepegit

@jepegit jepegit commented Sep 30, 2026

Copy link
Copy Markdown
Owner

Closes #1107. Epic M (#783) M4 — the cellpy side of ife-bat/batbase#474 (files[] on the journal API).

What

  • Contract — FileRef(kind, uri, order, size, mtime, checksum, loader, location); MetaRecord.files: tuple[FileRef, ...] = () (dicts coerced; raw_files() / cellpy_file() helpers); validate_record rejects unknown kinds, empty / duplicate URIs; ExternalLink.files records the URIs cellpy opened (key omitted from to_dict when empty → old meta.json documents unchanged).
  • Cell path — cellpy.get(source=, key=, kind=, project=, source_extra=, strict=) and CellpyCell.from_source(...). The record's pointers fill filename / cellpy_file / instrument only where the caller gave nothing, then the existing raw-vs-cellpy branch runs; no pointers ⇒ filefinder.search_for_files(cell_name) as today. The record is applied before explicit keywords, so mass= still wins. strict defaults to True when the source is the only way to find the data (unknown/unreachable raises; no record ⇒ NoDataFound), False when a filename is given (enrichment; missing record is a warning).
  • Batch path — batch.from_source(source, key, kind="tag", ...) (+ Batch.from_source, utils.batch shim) builds journal pages from the records via new cellpy/batch/source.py (pages_from_records, journal_from_records). Rows without pointers go through the same _dbengine.find_files call journal_from_db uses (file_search=False leaves them None). Back-links live in journal.session["external_links"] and are stamped onto loaded cells after update(); values are not re-applied, so journal precedence is intact.
  • Deferred — size/mtime short-circuit in update() (data is carried in raw_file_size / raw_file_mtime pages columns); cellpy-connectors adapter mapping BatBase files[] → FileRef (follow-up issue there).

Records without files behave exactly as before.

Docs

docs/guides/metadata_sources.md (step 5 + batch section), docs/agents/index.md, AGENTS.md, docs/api/readers.md, docs/api/batch.md, HISTORY [Unreleased]; design note .issueflows/04-designs-and-guides/metadata-sources.md ("File pointers (M4)"); test-registry rows.

Tests

  • tests/test_metadata_source_files.py (14) — contract, loads without filefinder (monkeypatched to raise), fallback parity, kwargs precedence, strict default, NoDataFound, save/load of ExternalLink.files, .cellpy pointer with summary refresh.
  • tests/test_batch_from_source.py (12) — pages parity, per-row file search, file_search=False, label de-dup, session round-trip, update() link stamping.
  • uv run pytest -m essential: 557 passed. Full suite: 1986 passed; 3 plotly/kaleido image-export timeouts are a local headless-Chromium problem, unrelated.
  • zensical build: no issues.

Pre-existing black --check drift in cellreader.py / facade.py / contract.py left untouched.

Made with Cursor

jepegit and others added 3 commits September 30, 2026 20:27
….get(source=) and batch.from_source (#1107)

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Epic M: consume file pointers from external metadata sources (skip filefinder)

1 participant