Decouple the database connection from module import

[alexevag]

Summary

importing ethopy.core.logger (and experiment, behavior) opens a live MySQL connection and declares tables as a side effect. So the package can't be imported, for tests or --help without a reachable database, and import can even block on a password prompt.

This proposes moving the connection out of import time into one explicit startup step. Logger still owns all DB communication, we only change when it connects.

The problem, with evidence

1. Connect on import. logger.py:78 calls _set_connection() at module top level → create_virtual_modules(SCHEMATA) → opens a real dj.Connection So import ethopy.core.logger connects to MySQL just by being loaded.
2. Prompt on import. With no password configured, that path calls getpass (helper_functions.py:80) → any import can block on stdin (headless/CI/cron).
3. Declare tables on import. experiment.py:29 imports the connected experiment module, then declares tables at module level (@experiment.schema / class Session, ~:997). Same in behavior.py:398+, which also has a cross-schema foreign key -> experiment.Trial (behavior.py:404).

These are chained: you can't import experiment without the connected experiment global, which only exists after _set_connection() ran.

My issue: unit tests need complex/fragile sys.modules patching done before import and are order-dependent (a test importing real datajoint first made test_logger hang), CI can't run without a DB, the docs build and CLI all trigger a connection they don't need.

How the fix works

The core idea: import should only define things, connecting is a separate step done once, on purpose.

Today the thing you import (experiment) is a live, connected object. To even have it, a connection must already have happened. The fix uses a deferred schema instead: an empty placeholder created with no connection.

# experiment.py
import datajoint as dj
schema = dj.Schema()        # empty box: no name, no connection, no DB

@schema
class Session(dj.Manual):
    definition = """ ... """
# ... Condition, Trial

Two things worth explaination:

• dj.Schema() does not know it is "experiment". The variable name and the filename mean nothing to DataJoint. The schema is nameless until you call activate("name"), the name comes only from that argument.
• Importing this creates nothing in MySQL. The classes are just Python objects. The actual CREATE DATABASE / CREATE TABLE happens inside activate(), using the connection.

A second schema can reference the first, the foreign key is just text until activation:

# behavior.py
from ethopy.core import experiment   # inert classes — importing does NOT connect

schema = dj.Schema()

@schema
class Activity(dj.Manual):
    definition = """
    -> experiment.Trial             # resolved at activate(), not now
    ...
    """

Then, in one place at startup (inside Logger / run.py:22), after the connection is open, you give each schema its name and create it:

conn = dj.Connection(...)                                   # the one connection
experiment.schema.activate(SCHEMATA["experiment"], connection=conn)
behavior.schema.activate(SCHEMATA["behavior"], connection=conn)  # after experiment

import defines, activate connects + creates. The connection lives in one place, not scattered across modules. And the same schema object is created empty at import and only filled in at activate() — Python runs a module body once, so nothing overwrites it later.

What we gain, and what it costs

Gain: you can import any module without a database. That is what unblocks tests, --help, linters, none of which want MySQL. Connection failures also become normal runtime errors instead of opaque ImportErrors, and import can no longer hang on a password prompt.

Cost: there is now an explicit activate() step that didn't exist before. Code that uses these schemas outside the normal app entry (an ad-hoc script, a notebook) must call activate() first, and forgetting it gives a "schema not activated" error instead of the old auto-connect-on-import magic.

Why it's still worth it: that cost is one call, in one place. The normal path (run.py → Logger) does the activation once, so experiments and end users see no new complexity, they import and run exactly as today. The only people who gain a one-line step are those importing the modules directly outside the app, and in exchange everyone gets DB-free imports. We trade a little explicit setup for removing a hidden, global side effect.

Using it outside the app

The app path is covered automatically: Logger activates once at startup, so anything going through it is already connected. For scripts/notebooks that query tables directly, the entry point is a single idempotent call:

import ethopy
ethopy.connect()        # opens the connection + activates the owned schemas

connect() live in one place (Logger / a top-level helper), the schema modules only define tables, they never check activation.

Plan

The change:

1. Convert the owned schemas (experiment, behavior) to deferred dj.Schema(); their table classes are defined at import with no connection.
2. Update cross-module imports to use those table classes directly instead of the connected globals from logger.
3. Remove the module-level _set_connection() at logger.py:78, do the connect + activate() once at startup, in Logger / run.py.

Open questions for review

Where should connect()/activate() live , Logger.__init__, a Logger.connect(), or run.py?