Epic M: skip raw-file stat on update() when the source's FileRef size/mtime match (#1124)

.issueflows/03-solved-issues/issue1124_original.md

@@ -0,0 +1,26 @@
+# Issue #1124: Epic M: use FileRef size/mtime from the metadata source to skip stat-ing raw files on update()
+
+Source: https://github.com/jepegit/cellpy/issues/1124
+
+## Original issue text
+
+## Context
+
+#1107 (PR #1119) added `FileRef(size, mtime, checksum, …)` on `MetaRecord.files`, and `batch.from_source` already carries them into the journal pages as `raw_file_size` / `raw_file_mtime`. Nothing consumes them yet: `CellpyCell.update()` / `Batch.refresh()` still `stat` every raw file (slow on `scp://`/SFTP shares) to decide whether anything changed.
+
+## Spec
+
+1. When a cell was loaded via a source record whose raw `FileRef`s carry `size` and/or `mtime`, store them next to the back-link (`ExternalLink` or `Data` provenance — pick one, document it) so they survive save/load.
+2. `update()` / `refresh()` / `poll()`: if the source's recorded `size`+`mtime` equal what cellpy already loaded, skip the remote `stat` and treat the file as unchanged. A changed value, or a missing one, falls back to today's `stat`-based path. `checksum` is informational only (no hashing of remote files).
+3. Batch: `Batch.refresh()` may consult `raw_file_size` / `raw_file_mtime` the same way; `batch.from_source(...).update()` followed by a second `from_source` fetch can short-circuit per cell.
+4. Never skip when the user forces a reload (`recalc=True`, `RAW_ONLY`, explicit `update(force=…)` if that exists).
+
+## Acceptance
+
+- A cell loaded from a `DictMetadataSource` record with `size`/`mtime` does not call the path `stat` on `update()` when the values match (monkeypatched `OtherPath.stat` asserts not called), and does when they differ.
+- Records without `size`/`mtime` behave exactly as today.
+- Round-trip: the recorded values survive `.cellpy` save/load.
+
+## Related
+
+#1107, #783 (Epic M), ife-bat/batbase#474 (`size`/`mtime` on `TestDataFile`), incremental-load protocol (#779 / #164).

.issueflows/03-solved-issues/issue1124_plan.md

@@ -0,0 +1,123 @@
+# Issue #1124 — plan
+
+Branch: `cursor/1124-fileref-skip-stat-ae40` (cloud-agent branch policy; the
+issue-flow `<N>-<slug>` form is not available here). Repo: `jepegit/cellpy`.
+
+## Goal
+
+A cell (or batch cell) that came from a metadata-source record carrying
+`FileRef.size` / `FileRef.mtime` skips the remote `stat` in
+`CellpyCell.update()` when those values equal what cellpy already loaded.
+Everything without such values behaves exactly as today.
+
+## Constraints
+
+- Follow [metadata-sources.md](../04-designs-and-guides/metadata-sources.md)
+  and [incremental-load-protocol.md](../04-designs-and-guides/incremental-load-protocol.md):
+  `ExternalLink.to_dict()` must stay byte-identical for documents that carry
+  no new data (omit the new key when empty); `raw` payloads never persisted.
+- `checksum` is informational only — no hashing of remote files.
+- `force=True` already bypasses change detection; keep that. `recalc` /
+  `RAW_ONLY` live on the `cellpy.get` / batch-load path, which this change
+  does not touch.
+- No cellpycore change; no new dependency.
+- KISS: one dataclass field, one comparison helper, one hook in
+  `_raw_sources_changed`. No new module.
+
+### Prior art
+
+- `CellpyCell._raw_sources_changed` / `_refresh_fid` (`readers/cellreader.py`)
+  — the stat-based check `update()` uses today; the hook goes here.
+- `ds.FileID.populate` (`readers/data_structures.py`) — `size` (int) and
+  `last_modified` (epoch float from `st_mtime`) are what "cellpy already
+  loaded" means. Coexist.
+- `MetaRecord.link(files=…)` / `ExternalLink.files` (`metadata_sources/contract.py`)
+  — the back-link that already records opened URIs; extend, do not replace.
+- `batch/source.py::pages_from_records` — already derives
+  `raw_file_size` / `raw_file_mtime` journal columns from the refs and builds
+  the link dicts stamped by `Batch._stamp_external_links`. Reuse: links built
+  there pick up the new field automatically.
+- `check_file_ids` (raw-vs-cellpy stat on first batch load) — same stats, but
+  a different path; out of scope here (see Open questions).
+- Toolbox (`00-tools/`): nothing relevant. Graph: `graphify-out/` absent.
+
+## Approach
+
+1. **Persist the hints on the back-link** (`contract.py`).
+   `ExternalLink.file_refs: tuple[FileRef, ...] = ()` — the record's `raw`
+   refs that carry `size` and/or `mtime`. `to_dict` writes `"file_refs"`
+   (list of `FileRef.to_dict()`) only when non-empty; `from_dict` reads it
+   back. `MetaRecord.link(files=…)` fills it from `raw_files()` restricted to
+   the given URIs (when `files` is empty: every raw ref with a stat). This is
+   the "pick one, document it" answer: the link, not `Data._provenance`,
+   because the link already is the per-source persisted object and
+   `_stamp_external_links` / `apply_meta_document` / `from_cell` all copy it.
+2. **Cell path** (`cellreader.py`): `_apply_meta_record(record, files=())`
+   unchanged in signature; when `files` is empty (a later `c.fetch_meta(...)`
+   on an already loaded cell) match the record's raw refs against the cell's
+   `raw_data_files` full names so a re-fetch refreshes the hints.
+3. **Change detection** (`cellreader.py`): in `_raw_sources_changed`, before
+   building `ds.FileID(fid.full_name)` ask `_source_file_hint(fid)` — scan
+   `data.external_links[*].file_refs` for a ref whose
+   `OtherPath(uri).full_path` equals `fid.full_name` (fallback: same `name`
+   when unique). If a hint exists and `_hint_matches_loaded(hint, fid)` is
+   true, log at debug (`"update: <name> unchanged per <source> record; stat
+   skipped"`) and treat that file as unchanged. Otherwise fall through to the
+   stat path exactly as today.
+   `_hint_matches_loaded`: every value the ref carries must match — `size`
+   as `int ==`; `mtime` via `_mtime_epoch(value)` (int/float epoch, or ISO
+   8601 through `datetime.fromisoformat`; naive → UTC) compared with
+   `fid.last_modified` within 1 s. A ref with neither value, an unparsable
+   mtime, or a `fid` without stats never matches.
+4. **Batch**: `pages_from_records` already calls `record.link(files=used)`, so
+   the session link dicts gain `file_refs`; `_stamp_external_links` restores
+   them onto loaded cells; `Batch.refresh()` / `poll()` call `c.update()` and
+   so skip the stat per cell. No facade change needed beyond a docstring
+   line. Re-fetching to refresh the hints is `c.fetch_meta(...)` (step 2) or
+   a new `batch.from_source(...)`; no extra API.
+5. **Docs**: new row in `metadata-sources.md` (decision + comparison rule);
+   one line in `docs/agents/index.md` and the `AGENTS.md` quick fact for
+   `c.update()`; `update()` docstring step 1. HISTORY entry at close.
+
+## Files to touch
+
+- `src/cellpy/readers/metadata_sources/contract.py` — `ExternalLink.file_refs`,
+  dict round trip, `MetaRecord.link` filling it.
+- `src/cellpy/readers/cellreader.py` — `_source_file_hint`,
+  `_hint_matches_loaded` (+ `_mtime_epoch` helper), hook in
+  `_raw_sources_changed`, `_apply_meta_record` re-fetch matching, docstring.
+- `src/cellpy/batch/facade.py` — `refresh` docstring note only.
+- `tests/test_metadata_source_files.py` (or new `tests/test_source_file_hints.py`
+  if it grows past ~120 lines) — see test strategy.
+- `.issueflows/04-designs-and-guides/metadata-sources.md`,
+  `docs/agents/index.md`, `AGENTS.md` — one row / one line each.
+- `.issueflows/01-current-issues/issue1124_status.md`.
+
+## Test strategy
+
+`uv run pytest tests/test_metadata_source_files.py tests/test_cell_update.py tests/test_batch_from_source.py tests/test_batch_live.py`
+then `uv run pytest -m essential`. Fixture: `DictMetadataSource` records with
+`FileRef(size=<res stat size>, mtime=<res stat mtime as ISO UTC>)`; monkeypatch
+`cellpy.internals.otherpath.OtherPath.stat` to raise `AssertionError`.
+
+- matching `size`+`mtime` → `c.update()` is `False`, `stat` not called.
+- `size` off by one → `stat` called (falls back; file unchanged → `False`).
+- `mtime` only, matching → skipped; `mtime` unparsable → `stat` called.
+- record without `size`/`mtime` → `stat` called (today's behaviour).
+- `update(force=True)` still reloads (existing test in `test_cell_update.py`
+  stays green).
+- round trip: `c.save()` → `cellpy.get(.cellpy)` keeps `file_refs`; old
+  `ExternalLink.to_dict()` without refs has no `"file_refs"` key.
+- batch: `Batch.from_source` session link carries `file_refs`; after
+  `update()` the loaded cell's link has them and `b.refresh()` skips `stat`.
+- re-fetch on a loaded cell (`c.fetch_meta`) replaces the hints.
+
+## Open questions
+
+- `check_file_ids` (raw-vs-cellpy stat on first batch load / `cellpy.get`
+  with a cellpy file) could use the same journal `raw_file_size` /
+  `raw_file_mtime`. Recommendation: separate follow-up issue; this one is
+  scoped to `update()` / `refresh()` / `poll()` as titled.
+- mtime tolerance 1 s and naive-ISO-means-UTC: recommended defaults; a
+  mismatch only costs one `stat`, never a wrong skip in the other direction
+  except a source whose clock is off by < 1 s.

.issueflows/03-solved-issues/issue1124_status.md

@@ -0,0 +1,45 @@
+# Issue #1124 — status
+
+- [x] Done
+
+Branch: `cursor/1124-fileref-skip-stat-ae40` (cloud-agent branch policy; not
+the `<N>-<slug>` form). Plan: `issue1124_plan.md` (accepted 2026-10-01,
+followed as written).
+
+## What's done
+
+- `ExternalLink.file_refs: tuple[FileRef, ...]` — raw refs with `size` /
+  `mtime`, restricted to opened URIs; `to_dict` emits `"file_refs"` only when
+  non-empty (pre-#1124 `meta.json` byte-identical); `file_ref_for(uri)`.
+  `MetaRecord.link(files=…)` fills it, so `cellpy.get(source=…)`,
+  `fetch_meta`, `pages_from_records` → `_stamp_external_links` all carry it.
+- `CellpyCell._raw_sources_changed`: `_source_file_hint(fid)` (URI /
+  `OtherPath.full_path` / unique basename) + module-level
+  `_source_hint_matches_loaded` (`size` int-equal, `mtime` epoch or ISO-8601
+  within `SOURCE_MTIME_TOLERANCE` = 1 s, naive ⇒ UTC; every carried value
+  must match) skips `ds.FileID(...)` (no `is_file` / `stat`) and logs at
+  debug. `force=True` unchanged.
+- `_apply_meta_record` on a loaded cell (re-fetch) keeps only refs pointing
+  at `raw_data_files`, so fresher source stats replace the hints.
+- Tests: `tests/test_source_file_hints.py` (17; 6 essential, registry rows
+  added). Related suites green: `test_metadata_source_files`,
+  `test_metadata_sources`, `test_cell_update`, `test_batch_from_source`,
+  `test_batch_live`, `test_live_poll` (88 passed).
+- Docs: `metadata-sources.md` new section; `docs/agents/index.md`,
+  `AGENTS.md`, `update()` and `Batch.refresh()` docstrings; HISTORY bullet.
+
+## Verification
+
+- `uv run pytest -m essential --ignore=tests/test_arbin_variants_two_stage.py --ignore=tests/test_load_since.py`:
+  953 passed, 74 skipped, 2 failed. Both failures
+  (`tests/test_filefinder.py::test_find_by_project_cellpy_range`,
+  `::test_find_by_project_otherpath_local`) fail identically on a clean
+  `master` checkout in this environment — not from this change. The two
+  ignored modules fail to import here (`pyodbc` needs `libodbc.so.2`), also
+  environmental.
+
+## Remaining work / follow-ups (not in scope)
+
+- `check_file_ids` (raw-vs-cellpy stat on first `cellpy.get` / batch load)
+  could consult journal `raw_file_size` / `raw_file_mtime` the same way —
+  separate issue.

.issueflows/04-designs-and-guides/metadata-sources.md

@@ -38,7 +38,21 @@ capacity, project without cellpy depending on one lab's API.
 | Back-link | `ExternalLink.files: tuple[str, ...]` = URIs cellpy opened because the record pointed at them; `to_dict` omits the key when empty so pre-#1107 `meta.json` documents are byte-identical. |
 | Batch path | `Batch.from_source(source, key, *, kind="tag", project, name, policy, file_search=True, file_search_kwargs, strict=True, **extra)`; module `batch.from_source` + `utils.batch` shim. `batch/source.py::pages_from_records` builds one row per record (`filename`/`label` = `test.cell_name` → `external_id` → `cell_NNN`, de-duplicated with `_2` suffixes; mass/area/loading/nom_cap/nom_cap_specifics/cycle_mode; `instrument` = first raw `loader`; `raw_file_names` / `cellpy_file_name`; `raw_file_size` / `raw_file_mtime` when known; `external_id`, `source_uri`). Rows without pointers ⇒ `_dbengine.find_files` (the `journal_from_db` call) on just those rows; `file_search=False` leaves `None` (#1017 rule). `project` defaults to the source name (no `project` field on `CellMeta`); `name` to `<source>_<kind>_<key>` slug. |
 | Batch provenance | Links kept in `journal.session["external_links"]` (`{label: ExternalLink.to_dict()}`, survives `write_journal`); `Batch.update()` → `_stamp_external_links()` copies them onto loaded cells. Values are **not** re-applied (journal precedence intact); per-field `Resolution.origin_of` stays a cell-path feature. |
-| Deferred | `size`/`mtime` short-circuit in `update()` / `refresh()` (data is carried, logic is a follow-up); cellpy-connectors adapter mapping BatBase `files[]` → `FileRef` (follow-up issue there). |
+| Deferred | cellpy-connectors adapter mapping BatBase `files[]` → `FileRef` (follow-up issue there). |
+
+## Stat skip on `update()` (#1124)
+
+| Topic | Decision |
+| --- | --- |
+| Where the hints live | `ExternalLink.file_refs: tuple[FileRef, ...]` — the record's **raw** refs that carry `size` and/or `mtime`, restricted to the URIs cellpy opened (`MetaRecord.link(files=…)`). Chosen over `Data._provenance` because the link is already the per-source persisted object that `_stamp_external_links`, `apply_meta_document` and `from_cell` copy. `to_dict` writes `"file_refs"` only when non-empty, so pre-#1124 `meta.json` stays byte-identical. `ExternalLink.file_ref_for(uri)` looks one up. |
+| Check | `CellpyCell._raw_sources_changed` asks `_source_file_hint(fid)` (URI equal to `fid.full_name`, or `OtherPath(uri).full_path` equal; else a *single* ref with the same basename) before building `ds.FileID(...)`. `_source_hint_matches_loaded(ref, fid)`: every value the ref carries must match — `size` as `int ==`, `mtime` as epoch (number, or ISO-8601 via `fromisoformat`; naive ⇒ UTC) within `SOURCE_MTIME_TOLERANCE` (1 s) of `fid.last_modified`. A ref with neither value, an unparsable mtime or a fid without stats never matches ⇒ stat as before. `force=True` bypasses the whole check (unchanged). `checksum` is never used. |
+| Re-fetch | `c.fetch_meta(source, key, kind=…)` on a loaded cell rebuilds the link; with no opened-URI list the refs are filtered to those pointing at `raw_data_files` (`_ref_points_at`), so the source's fresher `size`/`mtime` become the new hints. Batch: `pages_from_records` → session links → `_stamp_external_links` → `refresh()` / `poll()` benefit without facade changes; a new `batch.from_source(...)` is the batch-side re-fetch. |
+| Not done | `check_file_ids` (raw-vs-cellpy stat on the first `cellpy.get` / batch load) could consult the journal `raw_file_size` / `raw_file_mtime` the same way — separate follow-up. |
+
+Semantics: the source is the system of record for the file. While its
+recorded stats equal what cellpy loaded, cellpy trusts it and does not touch
+the share; a tester that keeps writing is noticed once the source re-scans
+(and cellpy re-fetches) or when `force=True`.
 
 Alternatives rejected: a new `Layer` for "source files" (files are not metadata; they select *what to load*); re-running `_apply_meta_record` per batch cell (would put the source above journal overrides); always writing `files` into `ExternalLink.to_dict()` (breaks byte-for-byte stability of old documents for no gain).
 

.issueflows/04-designs-and-guides/test-registry.md

@@ -275,6 +275,13 @@ current issue**. `/iflow-doctor` may audit the whole suite against this table.
 | tests/test_batch_from_source.py::test_from_source_builds_pages_like_a_journal | yes | | Batch.from_source | #1107 | |
 | tests/test_batch_from_source.py::test_update_loads_pointed_files_and_stamps_links | yes | | Batch.update → _stamp_external_links | #1107 | loads testdata |
 | tests/test_batch_from_source.py (other 7) | no | | naming, duplicates, file_search=False, session round-trip, strict/no-record errors | #1107 | offline |
+| tests/test_source_file_hints.py::test_mtime_epoch | yes | yes | cellreader._mtime_epoch | #1124 | offline, parametrized |
+| tests/test_source_file_hints.py::test_hint_matches_only_when_every_recorded_stat_agrees | yes | yes | cellreader._source_hint_matches_loaded | #1124 | offline |
+| tests/test_source_file_hints.py::test_link_keeps_stat_carrying_raw_refs_and_round_trips | yes | yes | MetaRecord.link / ExternalLink.file_refs dict round trip | #1124 | byte-identical old docs |
+| tests/test_source_file_hints.py::test_matching_hint_skips_stat | yes | yes | CellpyCell._raw_sources_changed hint path | #1124 | OtherPath.stat spy, loads testdata .res |
+| tests/test_source_file_hints.py::test_differing_size_falls_back_to_stat | yes | yes | hint mismatch → stat | #1124 | proves the spy sees stat |
+| tests/test_source_file_hints.py::test_record_without_stats_behaves_as_today | yes | yes | no-hint parity | #1124 | today's behaviour kept |
+| tests/test_source_file_hints.py (other 6) | no | | mtime-only, bad mtime, force, save/load, re-fetch, batch refresh | #1124 | uses testdata .res |
 
 **Columns**
 

AGENTS.md

@@ -388,6 +388,9 @@ Quick facts:
   `cellpy.get(source="batbase", key="SAL_010", kind="tag")` opens the
   record's file pointers (`filefinder` only as fallback);
   `batch.from_source("batbase", "SAL_010")` does the same for a whole tag.
+  Raw pointers with `size`/`mtime` stay on `external_links[src].file_refs`;
+  `update()` / `refresh()` skip the remote `stat` while they match what was
+  loaded (`force=True` or a differing value → stat/reload as usual).
 - Frames: `c.data.raw` / `.steps` / `.summary`; columns via `c.schema.*`.
   After a raw load, each cycle's raw capacity starts at 0. A forgotten tester
   reset that 1.x plotted as doubled capacity is rebased on load for every

HISTORY.md

@@ -72,6 +72,15 @@
   troubleshooting), wired into the guides index, How-do-I and `llms.txt`.
   (#1023)
 
+* Epic M: use `FileRef` size/mtime from the metadata source to skip stat-ing
+  raw files on `update()`. Raw pointers that carry `size` / `mtime` are kept
+  on `ExternalLink.file_refs` (persisted in v9 `meta.json`, omitted when
+  empty) and `CellpyCell.update()` / `Batch.refresh()` / `poll()` treat the
+  file as unchanged without a remote `stat` while they equal what was
+  loaded; a differing or missing value, or `force=True`, falls back to
+  today's path. A re-run of `fetch_meta` / `batch.from_source` refreshes the
+  hints. (#1124)
+
 ## [2.1.5.post6] - 2026-09-25
 
 * `summary_collector(...).plot()` keeps a lone charge or discharge series

docs/agents/index.md

@@ -204,6 +204,12 @@ Useful methods on `CellpyCell` (non-exhaustive):
   URIs used. Same as `CellpyCell.from_source(source, key, kind=)`.
   Batch: `batch.from_source("batbase", "SAL_010")` (kind `"tag"`) builds the
   journal pages from the records, then `b.update()`.
+  When a raw pointer carries `size` / `mtime`, it is kept on
+  `c.external_links[source].file_refs` and `c.update()` / `b.refresh()` /
+  `poll()` skip the remote `stat` while those values equal what was loaded
+  (a differing or missing value falls back to the stat; `force=True`
+  always reloads). Re-run `c.fetch_meta(...)` or `batch.from_source(...)`
+  to pick up fresher values from the source.
 - `save` / `to_csv` / Excel helpers — persist for the user's workflow
 
 Deeper shape docs: [Data structure](../fundamentals/data_structure.md).

src/cellpy/batch/facade.py

@@ -390,6 +390,10 @@ def refresh(self, labels: Sequence[str] | None = None, raise_errors: bool = Fals
         The combined-summary cache is cleared when any cell changed, so
         ``summaries`` / ``plot()`` / collectors see the new data.
 
+        Cells of a `from_source` batch whose record carried raw-file
+        ``size`` / ``mtime`` skip the remote ``stat`` while those values match
+        what was loaded (#1124); ``force=True`` reloads regardless.
+
         Args:
             labels: subset of cell labels (default: every loaded cell).
             raise_errors: re-raise a cell's ``update`` error instead of