Skip to content

Epic M: skip raw-file stat on update() when the source's FileRef size/mtime match (#1124) - #1126

Merged
jepegit merged 3 commits into
masterfrom
cursor/1124-fileref-skip-stat-ae40
Oct 1, 2026
Merged

jepegit merged 3 commits into
masterfrom
cursor/1124-fileref-skip-stat-ae40

Conversation

@jepegit

@jepegit jepegit commented Oct 1, 2026

Copy link
Copy Markdown
Owner

Closes #1124

What

CellpyCell.update() (and therefore Batch.refresh() / poll()) no longer stats a raw file when the metadata-source record that pointed at it recorded the same size / mtime cellpy loaded. Slow scp:// / SFTP shares are the motivation (#1107 carried the values; nothing consumed them).

How

  • ExternalLink.file_refs: tuple[FileRef, ...] — the record's raw refs that carry size and/or mtime, restricted to the URIs cellpy opened. MetaRecord.link(files=…) fills it; to_dict writes "file_refs" only when non-empty, so pre-existing meta.json documents are byte-identical. ExternalLink.file_ref_for(uri) looks one up. The link is already copied by _stamp_external_links, apply_meta_document and from_cell, so save/load and batch stamping come for free.
  • CellpyCell._raw_sources_changed asks _source_file_hint(fid) (URI equal to fid.full_name, or OtherPath(uri).full_path equal, else a single ref sharing the basename) before building ds.FileID(...). _source_hint_matches_loaded(ref, fid): every value the ref carries must match — size as int ==, mtime as epoch (number or ISO-8601 via fromisoformat; naive ⇒ UTC) within SOURCE_MTIME_TOLERANCE (1 s). A ref with neither value, an unparsable mtime or a fid without stats never matches → stat as before. force=True bypasses the check as it always did. checksum is never used.
  • _apply_meta_record on an already loaded cell (a re-fetch_meta) keeps only refs pointing at raw_data_files, so fresher stats from the source replace the hints. Batch: pages_from_records → session links → _stamp_external_links → refresh() benefits without facade changes; a new batch.from_source(...) is the batch-side re-fetch.
  • Docs: metadata-sources.md section, docs/agents/index.md, AGENTS.md, update() / Batch.refresh() docstrings, HISTORY bullet.

Not in scope (follow-up): check_file_ids (raw-vs-cellpy stat on the first cellpy.get / batch load) could consult the journal raw_file_size / raw_file_mtime the same way.

Tests

tests/test_source_file_hints.py (17 tests, 6 essential) with an OtherPath.stat spy:

  • matching size+mtime → update() is False, stat never called
  • size off by one / unparsable mtime / record without stats → stat called (today's behaviour)
  • mtime-only hint; force=True still reloads
  • hints survive .cellpy save/load; fetch_meta re-fetch replaces them
  • batch.from_source session link carries file_refs; after update() the cell has them and b.refresh() skips stat
uv run pytest tests/test_source_file_hints.py                      # 17 passed
uv run pytest tests/test_metadata_source_files.py tests/test_metadata_sources.py \
  tests/test_cell_update.py tests/test_batch_from_source.py \
  tests/test_batch_live.py tests/test_live_poll.py                 # 88 passed
uv run pytest -m essential --ignore=tests/test_arbin_variants_two_stage.py \
  --ignore=tests/test_load_since.py                                # 953 passed, 74 skipped, 2 failed

The two essential failures (tests/test_filefinder.py::test_find_by_project_cellpy_range, ::test_find_by_project_otherpath_local) fail identically on a clean master checkout in this VM; the two ignored modules cannot import pyodbc here (libodbc.so.2 missing). Both environmental, not from this change.

Open in Web Open in Cursor 

cursoragent and others added 3 commits October 1, 2026 15:42
…ize/mtime match the load

Raw FileRefs that carry size and/or mtime are kept on ExternalLink.file_refs
(persisted in v9 meta.json, omitted when empty so older documents are
unchanged). CellpyCell._raw_sources_changed consults them before building a
FileID: when every recorded value equals what was loaded (size exact, mtime
within 1 s, ISO-8601 or epoch), the file counts as unchanged and the remote
stat is skipped. A differing or missing value, an unparsable mtime, or
force=True falls back to today's path. Re-running fetch_meta on a loaded
cell refreshes the hints; batch links built by pages_from_records carry them
so Batch.refresh()/poll() benefit without facade changes.

Refs #1124

Co-authored-by: Jan Petter Maehlen <jepe@ife.no>
Changelog bullet, agent usage notes (docs/agents/index.md, AGENTS.md), the
metadata-sources design doc section recording where the hints live and the
comparison rule, and test-registry rows for the new essential tests.

Co-authored-by: Jan Petter Maehlen <jepe@ife.no>
Co-authored-by: Jan Petter Maehlen <jepe@ife.no>
@jepegit
jepegit marked this pull request as ready for review October 1, 2026 16:15
@jepegit
jepegit merged commit c238f40 into master Oct 1, 2026
7 checks passed
@jepegit
jepegit deleted the cursor/1124-fileref-skip-stat-ae40 branch October 1, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Epic M: use FileRef size/mtime from the metadata source to skip stat-ing raw files on update()

2 participants