feat: file pointers from external metadata sources — cellpy.get(source=) and batch.from_source (#1107) - #1119
Merged
Merged
Conversation
….get(source=) and batch.from_source (#1107) Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1107. Epic M (#783) M4 — the cellpy side of ife-bat/batbase#474 (
files[]on the journal API).What
FileRef(kind, uri, order, size, mtime, checksum, loader, location);MetaRecord.files: tuple[FileRef, ...] = ()(dicts coerced;raw_files()/cellpy_file()helpers);validate_recordrejects unknown kinds, empty / duplicate URIs;ExternalLink.filesrecords the URIs cellpy opened (key omitted fromto_dictwhen empty → oldmeta.jsondocuments unchanged).cellpy.get(source=, key=, kind=, project=, source_extra=, strict=)andCellpyCell.from_source(...). The record's pointers fillfilename/cellpy_file/instrumentonly where the caller gave nothing, then the existing raw-vs-cellpy branch runs; no pointers ⇒filefinder.search_for_files(cell_name)as today. The record is applied before explicit keywords, somass=still wins.strictdefaults to True when the source is the only way to find the data (unknown/unreachable raises; no record ⇒NoDataFound), False when a filename is given (enrichment; missing record is a warning).batch.from_source(source, key, kind="tag", ...)(+Batch.from_source,utils.batchshim) builds journal pages from the records via newcellpy/batch/source.py(pages_from_records,journal_from_records). Rows without pointers go through the same_dbengine.find_filescalljournal_from_dbuses (file_search=Falseleaves themNone). Back-links live injournal.session["external_links"]and are stamped onto loaded cells afterupdate(); values are not re-applied, so journal precedence is intact.size/mtimeshort-circuit inupdate()(data is carried inraw_file_size/raw_file_mtimepages columns); cellpy-connectors adapter mapping BatBasefiles[]→FileRef(follow-up issue there).Records without
filesbehave exactly as before.Docs
docs/guides/metadata_sources.md(step 5 + batch section),docs/agents/index.md,AGENTS.md,docs/api/readers.md,docs/api/batch.md, HISTORY[Unreleased]; design note.issueflows/04-designs-and-guides/metadata-sources.md("File pointers (M4)"); test-registry rows.Tests
tests/test_metadata_source_files.py(14) — contract, loads withoutfilefinder(monkeypatched to raise), fallback parity, kwargs precedence, strict default,NoDataFound, save/load ofExternalLink.files,.cellpypointer with summary refresh.tests/test_batch_from_source.py(12) — pages parity, per-row file search,file_search=False, label de-dup, session round-trip,update()link stamping.uv run pytest -m essential: 557 passed. Full suite: 1986 passed; 3 plotly/kaleido image-export timeouts are a local headless-Chromium problem, unrelated.zensical build: no issues.Pre-existing
black --checkdrift incellreader.py/facade.py/contract.pyleft untouched.Made with Cursor