Make processors compatible for dlclive gui update

_toc.yml

@@ -51,6 +51,7 @@ parts:
   - file: docs/software/mouse_task_test.md
 - caption: Dev - Software package Documentation
   chapters:
+  - file: docs/software_package/experiment_lifecycle
   - file: docs/software_package/active_sensing_task
   - file: docs/software_package/dlc_processor
 - caption: Experiments - Training protocol and parameters

dj_pipeline/gui_transfer/README.md

@@ -381,11 +381,34 @@ With camera prefix from `IMG_SRC` (default `Imagingsource`), a full session typi
 
 The server-side mirror is `vr4mice/actions/populate_rig.py` → `get_files_paths()`.
 
+### Multi-camera rigs
+
+Some rigs run more than one camera at once (currently 3). Each camera's timestamp/video file
+carries its camera index as a suffix directly after the keyword, with no underscore in between:
+
+| Role | Example filename | GUI key |
+|------|------------------|---------|
+| Camera timestamps (camera 3) | `TS_vr4mice_Testmouse_2023-02-22_2_CAMERA3.npy` | `camera_path` |
+| Video (camera 3) | `vr4mice_Testmouse_2023-02-22_2_VIDEO3.avi` | `video_path` |
+
+`camera_number_from_filename()` in `utils/session_files.py` extracts that index (`CAMERA\d+` /
+`VIDEO\d+`, case-insensitive). `find_related_files()` uses it so autofill stays consistent
+across a single camera's files:
+
+- Pick a numbered `camera_path`/`video_path` file by hand → the sibling of the *other* role is
+  constrained to the same camera number, instead of grabbing whichever camera sorts first.
+- Pick a file with no camera number of its own (DLC, PROC, teensy) → which camera to autofill is
+  ambiguous, so it defaults to `DEFAULT_CAMERA_NUMBER` (currently `3`) rather than camera 1.
+- Single-camera rigs (no `CAMERA`/`VIDEO` suffix anywhere) are unaffected — this logic only
+  kicks in once more than one camera number is present for a session.
+
 ### How the GUI classifies files
 
 1. **Validation** — `modules/transfer.py` → `_set_path_format()` (glob patterns for the file picker).
 2. **Type tag** — `get_type()` scans for keywords: `VIDEO`, `TS`, `DLC`, `PROC`; otherwise `teensy_path`.
-3. **Sibling search** — `find_related_files()` lists configured rig folders and keeps files whose stem matches the selected session.
+3. **Sibling search** — `find_related_files()` lists configured rig folders, keeps files whose
+   stem matches the selected session, and (see *Multi-camera rigs* above) disambiguates by
+   camera number when more than one camera's files are present.
 
 ### If formats change
 
@@ -395,14 +418,15 @@ The server-side mirror is `vr4mice/actions/populate_rig.py` → `get_files_paths
 | Different date format | Stem parsing, auto-fill | Enter mouse/date/attempt manually |
 | New file category | Not shown in transfer section | Requires new GUI key + populate path |
 | Mouse names with `_` | Wrong stem split | Avoid underscores in mouse names or update regex |
+| Rig's default camera count/index changes | Ambiguous DLC/PROC/teensy pick defaults to the wrong camera | Update `DEFAULT_CAMERA_NUMBER` in `utils/session_files.py`, or select camera/video files by hand |
 
 ### Code to update (checklist)
 
 When changing rig naming, edit **together**:
 
 | File | What to change |
 |------|----------------|
-| `gui_transfer/utils/session_files.py` | `SESSION_RE`, validation helpers |
+| `gui_transfer/utils/session_files.py` | `SESSION_RE`, `CAMERA_NUMBER_RE`, `DEFAULT_CAMERA_NUMBER`, validation helpers |
 | `gui_transfer/modules/transfer.py` | `_set_path_format()`, `get_type()`, transfer keys |
 | `vr4mice/actions/populate_rig.py` | `get_files_paths()` |
 | `tests/unit/test_gui_transfer.py` | Golden filename examples |

dj_pipeline/gui_transfer/modules/transfer.py

@@ -14,6 +14,7 @@
 from utils.utils import check_files
 from utils.session_files import (
     PATH_KEYS_FOR_SEARCH,
+    camera_number_from_filename,
     dataset_stem_from_filename,
     find_related_files,
 )
@@ -89,7 +90,8 @@ def _set_labels():
 def _path_is_remote(key):
     """
     Determines whether the specified file type is expected to have a remote path or not.
-    Currently, it's the case only of video_path-typed file
+    Currently, it's the case only of video_path-typed file. In this module,
+    "remote" means the file is not part of the transfer/move set (it stays on rig).
 
     Args:
       key (str): The file type (key) to check.
@@ -211,8 +213,10 @@ def get_transfer_files(self, key=None, send=False):
         Returns:
           dict or None: The transfer file for the specified key, or all transfer files.
         """
-        if key is not None and key in self.get_keys():
-            return self.transfer_file[key]
+        if key is not None:
+            if key in self.transfer_file:
+                return self.transfer_file[key]
+            return None
 
         if send is True:
             ret = dict()
@@ -226,10 +230,17 @@ def get_transfer_files(self, key=None, send=False):
     def get_processed_files(self):
         """
         Get files that should be moved to processed_path after a successful submit.
+
+        Every file that was actually transferred (i.e. not remote-only, see
+        _path_is_remote) moves to processed_path once submit succeeds -
+        that's teensy/dlc/camera/proc plus the GUI-generated gui_output.
+        video_path is excluded: videos stay on the rig, they're never
+        transferred, so there's nothing to move.
         """
         ret = list()
-        for key in ("gui_output", "teensy_path"):
-            info = self.get_transfer_files(key=key)
+        for key, info in self.transfer_file.items():
+            if _path_is_remote(key):
+                continue
             if info:
                 ret.append(info)
         return ret
@@ -402,16 +413,25 @@ def _check_video(self, keys, video_label="video_path"):
     def _pre_fetch_files(self, filenames, skip_path=None):
         """
         Find sibling session files across configured rig folders.
+
+        On multi-camera rigs, the picked file's camera index (if any, e.g.
+        "..._CAMERA3.npy" / "..._VIDEO3.avi") constrains which sibling
+        camera/video file gets auto-filled, so it matches the camera the
+        user actually selected instead of whichever camera sorts first.
         """
         dataset_stem = dataset_stem_from_filename(filenames)
         if not dataset_stem:
             logger.warning(f"Could not parse session from filename: {filenames}")
             return []
 
+        camera_number = camera_number_from_filename(filenames)
+
         path_by_key = {
             path_key: config.get_path(path_key) for path_key in PATH_KEYS_FOR_SEARCH
         }
-        related = find_related_files(dataset_stem, path_by_key, get_type)
+        related = find_related_files(
+            dataset_stem, path_by_key, get_type, camera_number=camera_number
+        )
         skip_resolved = Path(skip_path).resolve() if skip_path else None
 
         processed_keys = list()

dj_pipeline/gui_transfer/utils/session_files.py

@@ -9,6 +9,11 @@
 from pathlib import Path
 
 SESSION_RE = re.compile(r"([A-Za-z0-9]+)_(\d{4}-\d{2}-\d{2})_(\d+)")
+CAMERA_NUMBER_RE = re.compile(r"(?:CAMERA|VIDEO)(\d+)", re.IGNORECASE)
+
+# Rig always has exactly 3 cameras; this is the camera to default to when a
+# picked file (e.g. DLC output) carries no camera number of its own.
+DEFAULT_CAMERA_NUMBER = 3
 
 PATH_KEYS_FOR_SEARCH = (
     "teensy_path",
@@ -30,6 +35,18 @@ def dataset_stem_from_filename(filename):
     return f"{match.group(1)}_{match.group(2)}_{match.group(3)}"
 
 
+def camera_number_from_filename(filename):
+    """
+    Extract the camera index from a rig filename (e.g. "..._CAMERA3.npy" or
+    "..._VIDEO3.avi" -> 3). Returns None for filenames with no camera suffix
+    (single-camera rigs, or non-camera files like DLC/PROC/teensy).
+    """
+    match = CAMERA_NUMBER_RE.search(Path(filename).stem)
+    if not match:
+        return None
+    return int(match.group(1))
+
+
 def parse_session_from_filename(filename):
     """
     Parse mouse name, attempt, and date from a filename.
@@ -85,22 +102,34 @@ def check_file_format(key, filename, format_spec, current_mouse=None):
     return mouse_name, attempt, date
 
 
-def find_related_files(dataset_stem, path_by_key, get_type_fn):
+def find_related_files(dataset_stem, path_by_key, get_type_fn, camera_number=None):
     """
     Find one file per transfer type that belongs to the same session.
 
     Args:
         dataset_stem: e.g. Testmouse_2023-02-22_2
         path_by_key: mapping config key -> directory path string
         get_type_fn: callable(filename) -> transfer key string
+        camera_number: if set, on a rig with multiple cameras (files
+            suffixed "..._CAMERA3.npy" / "..._VIDEO3.avi"), only match
+            candidate files for that camera index. Files with no camera
+            suffix (single-camera rigs, DLC/PROC/teensy) are unaffected.
+            If None (the file that was picked has no camera suffix, e.g.
+            DLC/PROC/teensy), a role with several different camera numbers
+            present is ambiguous; DEFAULT_CAMERA_NUMBER is used to resolve
+            it. If DEFAULT_CAMERA_NUMBER isn't among the candidates, there is
+            no safe default to autocomplete to, so that role is left out of
+            the result entirely (no fallback to e.g. the max camera number).
+            A role where every match shares the same number (or none has a
+            number at all) is unaffected.
 
     Returns:
         dict mapping transfer key -> Path
     """
     if not dataset_stem:
         return {}
 
-    found = {}
+    candidates = {}
     seen_dirs = set()
 
     for path_key in PATH_KEYS_FOR_SEARCH:
@@ -121,8 +150,28 @@ def find_related_files(dataset_stem, path_by_key, get_type_fn):
                 continue
             if dataset_stem_from_filename(filepath.name) != dataset_stem:
                 continue
+            file_camera_number = camera_number_from_filename(filepath.name)
+            if (
+                camera_number is not None
+                and file_camera_number is not None
+                and file_camera_number != camera_number
+            ):
+                continue
             file_key = get_type_fn(filepath.name)
-            if file_key not in found:
-                found[file_key] = filepath
+            candidates.setdefault(file_key, []).append((file_camera_number, filepath))
+
+    found = {}
+    for file_key, matches in candidates.items():
+        if camera_number is None:
+            distinct_numbers = {n for n, _ in matches if n is not None}
+            if len(distinct_numbers) > 1:
+                if DEFAULT_CAMERA_NUMBER not in distinct_numbers:
+                    continue
+                matches = [m for m in matches if m[0] == DEFAULT_CAMERA_NUMBER]
+        else:
+            exact = [m for m in matches if m[0] == camera_number]
+            if exact:
+                matches = exact
+        found[file_key] = matches[0][1]
 
     return found

docs/software/install_dj_pipeline.md

@@ -539,6 +539,12 @@ The rig GUI and **`populate_rig`** on the server assume the same session filenam
 
 Classification uses keyword tags (`TS`, `DLC`, `VIDEO`, `PROC`) and glob patterns in `gui_transfer/modules/transfer.py`; parsing lives in `gui_transfer/utils/session_files.py`. The server mirror is `vr4mice/actions/populate_rig.py` → `get_files_paths()`.
 
+**Multi-camera rigs:** on rigs with more than one camera, timestamp/video filenames carry a
+camera index suffix (`..._CAMERA3.npy`, `..._VIDEO3.avi`). `camera_number_from_filename()` +
+`find_related_files()` in `gui_transfer/utils/session_files.py` keep sibling autofill on the same
+camera; when the picked file has no camera number of its own (DLC/PROC/teensy) and several
+cameras are present, autofill defaults to `DEFAULT_CAMERA_NUMBER`. Details: `dj_pipeline/gui_transfer/README.md` → *Multi-camera rigs*.
+
 **If naming changes**, update GUI + populate + tests in one change set — patterns are **not** configurable in `config.json`. Full checklist and limitations: `dj_pipeline/gui_transfer/README.md` → *Rig filename contract*.
 
 Further GUI module details: `dj_pipeline/gui_transfer/README.md`.