Add np_sync module for VR-to-NP barcode alignment and tests

[CeliaBenquet]

• Update cron scenario workflow.
• change to PROBE barcode.
• Test once vr barcode is properly populated.
• Test population works

This pull request introduces a new pipeline for aligning VR (behavior) time to Neuropixels (NP) native time using shared barcode events, along with documentation and unit tests. The main changes include adding the barcode alignment fitting logic, a DataJoint table for storing and applying the alignment, and updating configuration and documentation to support and describe the new functionality.

Barcode alignment pipeline for VR-to-NP time synchronization:

• Added a new module vr4mice/analysis/np_sync.py implementing the barcode alignment logic, including a dataclass BarcodeAlignmentFit and the align_barcodes function to fit a linear mapping and interpolator from VR to NP time using shared barcode events.
• Added a new DataJoint computed table BarcodeSync in vr4mice/schema/np_sync.py to store the fitted alignment for each dataset/recording/DAQ, with methods to convert VR times to NP times using either the interpolator or the linear fit. This table handles missing data gracefully and logs errors.

Configuration and integration:

• Updated .env.compose.example and docker-compose.yml to mount the np_pipeline source directory into the container, ensuring that the VR pipeline can import NP schema definitions for alignment.

Documentation:

• Extended the DataJoint schema documentation (docs/software/datajoint.md) to include the new barcode alignment table and describe its dependencies and usage.

Testing:

• Added unit tests for the barcode alignment logic in test_np_sync.py, covering correct recovery of known fits, handling of missing values, skipping of unreliable early events, and correctness of the interpolator.This PR introduces a new utility for aligning VR and Neuropixels (NP) event streams using shared barcode values, along with its configuration and unit tests. The main addition is a function that computes a linear mapping and interpolator from VR event times to NP event times based on shared barcodes, addressing issues with unreliable early events. The changes also include Docker configuration updates to ensure the necessary code is available in the container.

---

to do to deploy:

• Add the path to that repo in the env.compose: replace NP_PIPELINE_SRC_PATH=./np_pipeline_stub

[CeliaBenquet]

I could populate for a given entry and use the align_timepoints method cleanly, I will drop before deployement.
Also tested for a session without NP and it seems to work.

@lecriste @arturoptophys this will require to setup the 2 repos similarly on server 3 as well, any things in mind that needs to be done. Note that I also added a DJ_PASS to the VR env files, as the naming differs between both, I don't know if that's necessary?

[lecriste]

The overall approach is sound and well-isolated (np_pipeline missing only skips this mode). Nothing blocking except the None-input bug in align_timepoints.

Findings:

1. align_timepoints crashes on None inputs vs docstring

File: dj_pipeline/vr4mice/schema/np_sync.py:568 — correctness

align_timepoints() promises "None entries pass through as None" but np.array(..., dtype=np.float64) raises TypeError on None in modern numpy — the promised no-neural-data code path breaks.

Failure scenario: Any caller doing BarcodeSync.align_timepoints(key, [1.0, None, 2.0]) — the exact "skip missing timestamps" pattern the docstring advertises — hits TypeError: float() argument must be a string or a real number, not 'NoneType' at np.array(timepoints, dtype=np.float64) on numpy≥1.24, never reaching the float(tx) if not np.isnan(tx) else None filter. Fix: coerce None→NaN before np.array (e.g. [np.nan if t is None else t for t in timepoints]).

---

2. barcode_overlap ZeroDivision when np_values empty

File: dj_pipeline/vr4mice/schema/np_sync.py:523 — correctness

len(fit.shared_barcodes) / len(np_values) raises ZeroDivisionError when NP fetched zero events; caught by the broad try/except and written to FailedSession with an opaque / by zero error message instead of a real reason.

Failure scenario: A key survives key_source (candidate had np_event_count>0 at query time) but by the time make() fetches ProbeBarcodeExtraction.Event & np_key the rows are absent/deleted → np_values has length 0 → ZeroDivisionError → FailedSession row division by zero with no operator-actionable reason. Guard on len(np_values) == 0 up front and record a clear "no NP events for key" reason instead.

---

3. Docs cite OneBoxDaq tables; code uses probe tables

File: docs/software/datajoint.md:622 — docs-mismatch

The DataJoint docs entry for BarcodeSync says it depends on acquisition.OneBoxDaq and reads from OneBoxBarcodeExtraction.Event, but the actual schema references ProbeBarcodeExtraction and acquisition.RecordingProbe. Anyone following the docs to trace lineage or write downstream code will look up nonexistent tables.

---

4. 'selected is None' branch unreachable via key_source

File: dj_pipeline/vr4mice/schema/np_sync.py:460 — dead-code

key_source is dj.U(*self.primary_key) & _candidate_relation(), so every key make() receives already has at least one row in _candidate_relation() & key; the if selected is None FailedSession branch can only fire on a race between key enumeration and fetch, which the surrounding try/except would already catch. The "skipped cleanly with FailedSession" story in the module docstring is thus mostly aspirational.

Failure scenario: Docstring at top of the file states "Datasets with no strict NP match are recorded in vr4mice.FailedSession and skipped cleanly." In practice, populate() never visits datasets without a match (key_source excludes them), so FailedSession stays empty for that case and there is no signal to the operator that VR-only sessions exist. Either document that VR-only datasets are simply invisible to the table, or extend key_source to include them (and have make() record "no NP match" on the FailedSession path).

[CeliaBenquet]

@lecriste before we deploy it, you will need to change some env files in server 3 fyi. lmk how you want to proceed.

[lecriste]

@lecriste before we deploy it, you will need to change some env files in server 3 fyi. lmk how you want to proceed.

OK, please list the changes in the PR description and I will apply them.

PS: I see only one new var added in https://github.com/MMathisLab/FreelyMovingVR4Mice/blob/6d260d9df657cd93ef20d3cba7d5c8a2341b465c/dj_pipeline/.env.compose.example

[CeliaBenquet]

• Make sure you have https://github.com/SCENE-Collaboration/FreelyMoving_Mouse_NP cloned on server 3
• Add the path to that repo in the env.compose: replace NP_PIPELINE_SRC_PATH=./np_pipeline_stub

I think that should be it

[lecriste]

• Make sure you have https://github.com/SCENE-Collaboration/FreelyMoving_Mouse_NP cloned on server 3

  • Add the path to that repo in the env.compose: replace NP_PIPELINE_SRC_PATH=./np_pipeline_stub

I think that should be it

Thanks! So only one env change, in .emv.compose
Can you please log the second bullet in the PR description?

[arturoptophys]

few minor things to consider, but can be addressed later

• maybe replace the rigid DEFAULT_SKIP_FIRST_N_BARCODES with just dropping the repeated readings of same barcode? cause 10 barcodes is 55 sec in worst case, which could be more than one trial.
• quality gate for BarcodeSync: only insert if franction matched is larger e,g, >.95? and at least x number of barcodes? extreme case of 2 matched barcodes producing a perfect r2=1 fit.
• docs/software/datajoint.md:424 still misstates the dependencies
• maintenance._schema_pairs() has no np_sync entry

[lecriste]

One new finding:

PYTHONPATH and sys.path fallback point to stale mount path

docker-compose.yml sets PYTHONPATH=/np_pipeline/src:${PYTHONPATH} and the schema fallback does sys.path.insert(0, "/np_pipeline"), but the actual bind mount is at /app/np_pipeline. Neither the env var nor the fallback references an existing directory in the container.

Failure scenario: Normal runs succeed only because working_dir: /app puts /app on sys.path implicitly, so from np_pipeline.schemas import ... resolves via /app/np_pipeline. But any invocation with a different CWD (cron/systemd entry, cd /tmp && python -m ..., a subprocess with cwd= set elsewhere) drops /app from sys.path — the initial import raises ModuleNotFoundError, the fallback inserts /np_pipeline (nonexistent), retries, and raises again. PYTHONPATH=/np_pipeline/src never helps. Change PYTHONPATH to include the actual mount (/app/np_pipeline) and update the fallback to the same, or drop both and document that np_pipeline is discovered relative to /app.

[lecriste]

LGTM

[CeliaBenquet]

maybe replace the rigid DEFAULT_SKIP_FIRST_N_BARCODES with just dropping the repeated readings of same barcode? cause 10 barcodes is 55 sec in worst case, which could be more than one trial.

as discussed, we should potentially remove game time duplicate, but this is fine for now :)