Epic M: skip raw-file stat on update() when the source's FileRef size/mtime match (#1124) - #1126
Merged
Merged
Conversation
…ize/mtime match the load Raw FileRefs that carry size and/or mtime are kept on ExternalLink.file_refs (persisted in v9 meta.json, omitted when empty so older documents are unchanged). CellpyCell._raw_sources_changed consults them before building a FileID: when every recorded value equals what was loaded (size exact, mtime within 1 s, ISO-8601 or epoch), the file counts as unchanged and the remote stat is skipped. A differing or missing value, an unparsable mtime, or force=True falls back to today's path. Re-running fetch_meta on a loaded cell refreshes the hints; batch links built by pages_from_records carry them so Batch.refresh()/poll() benefit without facade changes. Refs #1124 Co-authored-by: Jan Petter Maehlen <jepe@ife.no>
Changelog bullet, agent usage notes (docs/agents/index.md, AGENTS.md), the metadata-sources design doc section recording where the hints live and the comparison rule, and test-registry rows for the new essential tests. Co-authored-by: Jan Petter Maehlen <jepe@ife.no>
Co-authored-by: Jan Petter Maehlen <jepe@ife.no>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1124
What
CellpyCell.update()(and thereforeBatch.refresh()/poll()) no longerstats a raw file when the metadata-source record that pointed at it recorded the samesize/mtimecellpy loaded. Slowscp:/// SFTP shares are the motivation (#1107 carried the values; nothing consumed them).How
ExternalLink.file_refs: tuple[FileRef, ...]— the record's raw refs that carrysizeand/ormtime, restricted to the URIs cellpy opened.MetaRecord.link(files=…)fills it;to_dictwrites"file_refs"only when non-empty, so pre-existingmeta.jsondocuments are byte-identical.ExternalLink.file_ref_for(uri)looks one up. The link is already copied by_stamp_external_links,apply_meta_documentandfrom_cell, so save/load and batch stamping come for free.CellpyCell._raw_sources_changedasks_source_file_hint(fid)(URI equal tofid.full_name, orOtherPath(uri).full_pathequal, else a single ref sharing the basename) before buildingds.FileID(...)._source_hint_matches_loaded(ref, fid): every value the ref carries must match —sizeasint ==,mtimeas epoch (number or ISO-8601 viafromisoformat; naive ⇒ UTC) withinSOURCE_MTIME_TOLERANCE(1 s). A ref with neither value, an unparsable mtime or a fid without stats never matches → stat as before.force=Truebypasses the check as it always did.checksumis never used._apply_meta_recordon an already loaded cell (a re-fetch_meta) keeps only refs pointing atraw_data_files, so fresher stats from the source replace the hints. Batch:pages_from_records→ session links →_stamp_external_links→refresh()benefits without facade changes; a newbatch.from_source(...)is the batch-side re-fetch.metadata-sources.mdsection,docs/agents/index.md,AGENTS.md,update()/Batch.refresh()docstrings, HISTORY bullet.Not in scope (follow-up):
check_file_ids(raw-vs-cellpy stat on the firstcellpy.get/ batch load) could consult the journalraw_file_size/raw_file_mtimethe same way.Tests
tests/test_source_file_hints.py(17 tests, 6 essential) with anOtherPath.statspy:size+mtime→update()isFalse,statnever calledstatcalled (today's behaviour)force=Truestill reloads.cellpysave/load;fetch_metare-fetch replaces thembatch.from_sourcesession link carriesfile_refs; afterupdate()the cell has them andb.refresh()skipsstatThe two essential failures (
tests/test_filefinder.py::test_find_by_project_cellpy_range,::test_find_by_project_otherpath_local) fail identically on a cleanmastercheckout in this VM; the two ignored modules cannot importpyodbchere (libodbc.so.2missing). Both environmental, not from this change.