Recover/sync local mice/exp with parent DB

[maryapp]

Summary

• Adds run_base.py recovery/sync tooling to rebuild local mice / exp from GUI .npy, sync the mice registry from parent via host mysqldump (mysql.mk), and populate base.Base after recover.
• Hardens multi-user Docker ops (UID/GID exec, compose fallbacks, latin1 session_notes, water-restriction _→__ FK mirror) and logs the live MySQL target before destructive modes.
• Production use: this branch was run on the production rig (server3) under the Linux account master to restore local base/exp/mice after an outdated local registry — recover, day checks/fixes, and orphan cleanup against the local DB only (not main).

The diff looks big because it includes the recovery tool, the sync rewrite, day repair, DJ2 workarounds, and ops hardening that were needed to actually run this in production — not because every file is a new feature forever, + tests

Test plan

• On a rig with local Docker DB: make -f mysql.mk sync-mice-from-main then python run_base.py recover_base inside make bash
• Confirm base.Base grows after recover; Dataset−Base gap is only Latencytest* / known failures
• python run_base.py check_session_days (and fix_session_days dry-run / --force if needed)
• Dry-run cleanup_orphans / cleanup_mice and verify the log shows DJ_HOST + live host:port (not DJ_MAIN_HOST)
• Re-run recover for SessionScoreSheet cases that previously failed on __water_restriction FK

[maryapp]

cc @arturoptophys, @lecriste exp and mice/base have been updated, again, I don't like the idea of doing it manually (aka sync - change days. move .npy files from processed, so here is the code that takes care of it)

[maryapp]

Note (DataJoint _ vs __ — upstream naming change)

We have MySQL tables with a double underscore (__), e.g. mouse_score_sheet__water_restriction. Newer DJ (incl. DJ2) effectively moved to a single underscore (_), e.g. mouse_score_sheet_water_restriction, and the __ form is deprecated / no longer what inserts target.

That mismatch breaks local DBs that still have FKs (notably exp.session_score_sheet → water restriction) pointing at the old __ table, while DataJoint and mysqldump sync write into _. Result: SessionScoreSheet FK failures even though the _ row “exists.”

This branch works around that DJ naming change by mirroring (mouse_name, doc) into __ when needed during recover/populate, and by mapping _↔ in the host mice sync dump load. (cc @lecriste to investigate the bug)

[maryapp]

Why not DataJoint dual-connect

A Python sync_mice used to connect to main, fetch the mice registry, then
switch back to local and upsert — all in one DataJoint process. That was
dropped (ce2d583, 97d3641) because DataJoint 2.x does not reliably support
two MySQL endpoints in one process.
dj.conn() is a singleton. Imported mice / exp Schema objects keep
their own .connection. dj.conn(reset=True) often left the TCP session and
the schema objects on the previous server.

Well... @lecriste, @arturoptophys, happy to know how and why it worked for you for mouse hockey, may it be the case since it's 2 diff servers?...

What actually failed:

1. TLS — main is plain TCP. use_tls=False in config was ignored; DJ still
   SSL-wrapped and crashed.
2. Same IP, two ports — local :3309 vs main :3306. Changing host without
   port (or trusting config after reset) still queried local.
3. Sticky TCP — after “restore”, config said :3309 but live @@port was
   still 3306.
4. Sticky schemas — mice.Mouse() stayed on the previous session. Sync
   could count local rows while dj.conn() was on main — or write to main.
5. Wrong upserts even on the right host — case-sensitive names; parent
   __ part tables vs local _; missing Strain / SurgeryType FKs; replace=True
   on Strain deleted a parent PK that local Mouse already referenced.

Safeguards (CONNECTION_ID(), refuse local endpoint, verify @@port) mostly
proved the switch was still wrong. Dual-connect was unsafe in both directions:
read the wrong DB, or write to main.

run_base.py no longer has sync_mice. Incomplete mice warn you to run the
host mysql target instead.

Dual DataJoint (Python, one process)

Pros

• Same stack as the rest of the pipeline (run_base.py, table classes, types).
• Can filter incomplete local mice in Python before fetch.
• No host mysql / mysqldump required.
• One command inside make bash.

Cons

• DJ2 singleton + schema objects do not support two DBs.(?? is it true)
• Easy to read/write the wrong server (including main).
• SSL, port, and rebind bugs are opaque.
• replace=True vs FKs is easy to get wrong.
• Hard to test; failed in production after many patches.

MySQL bash (mysqldump → mysql)

Current path: make -f mysql.mk sync-mice-from-main →
scripts/sync_mice_mysql.sh. Two separate mysql clients, no DJ singleton.

Pros

• No shared session — dump from DJ_MAIN_*, load into local DJ_HOST.
• Never deletes on main (dump only).
• Explicit: INSERT IGNORE for lookups, REPLACE for manuals,
  FOREIGN_KEY_CHECKS=0.
• Rewrites __ → _ when table names differ.
• Identifies servers by @@server_uuid, not @@port (Docker’s internal port
  is often 3306).
• Optional filter: MOUSE=Name1,Name2.
• Easy to inspect the SQL dump.

Cons

• Host-side only (mysql + mysqldump on PATH), not inside make bash.
• Bypasses DataJoint (no DJ validation, lineage, or Python types).
• REPLACE can overwrite local Mouse / score-sheet rows.
• Table list and __↔_ mapping live in the script.
• Credentials in .env; dump files briefly on disk.
• Replication still matters if local is a replica.

Bottom line: dual DJ looked cleaner in code but was unsafe because DJ2 would
not reliably leave main. Bash is dumber and split across host vs container, but
it copies the right rows to the right DB.

cc @arturoptophys, so to consider what approach to use... I feel like mysql is more stable

[maryapp]

version from vr4mice docker on server3: datajoint==2.2.4

[maryapp]

TODOS: (cc @lecriste)

• consider how we sync (Dual DJ VS Mysql)
• sync back exp/mice to (-aux + auto replication to Main)
• have the sync in cronjob (the current one)
• change path in fetch_menu (since generate )