feat: NWB export backend (schema + handler + cronjobs)

notebooks/nwb_production_demo.ipynb

@@ -0,0 +1,284 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "bd934085",
+   "metadata": {},
+   "source": [
+    "# NWB Production Schema Demo\n",
+    "\n",
+    "This notebook demonstrates how to import and use the NWB Production schema from the U19 pipeline.\n",
+    "\n",
+    "The NWB Production schema provides tables for:\n",
+    "- Managing NWB export jobs\n",
+    "- Tracking job status through the pipeline\n",
+    "- Logging status transitions\n",
+    "- Storing validation results"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "87b8f451",
+   "metadata": {},
+   "source": [
+    "## 1. Import Required Modules\n",
+    "\n",
+    "First, import the necessary modules including DataJoint and the U19 pipeline."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "1df04df1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[2026-01-17 13:18:24,331][INFO]: DataJoint 0.14.4 connected to ct5868@datajoint00.pni.princeton.edu:3306\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ NWB Production schema imported successfully!\n",
+      "Schema name: u19_nwb_production\n"
+     ]
+    }
+   ],
+   "source": [
+    "# from scripts.conf_file_finding import try_find_conf_file\n",
+    "\n",
+    "# try_find_conf_file()\n",
+    "\n",
+    "import datajoint as dj\n",
+    "import pandas as pd\n",
+    "from datetime import datetime\n",
+    "\n",
+    "# Import the NWB production schema\n",
+    "from u19_pipeline import nwb_production\n",
+    "\n",
+    "print(\"✅ NWB Production schema imported successfully!\")\n",
+    "print(f\"Schema name: {nwb_production.schema.database}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "23481c44",
+   "metadata": {},
+   "source": [
+    "## 2. Explore NWB Export Status Table\n",
+    "\n",
+    "The `NwbExportStatus` table is a lookup table containing the possible status values for NWB export jobs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "afe8eb9a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Display the table structure\n",
+    "nwb_production.NwbExportStatus()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "25b38773",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# View all status values\n",
+    "status_df = pd.DataFrame(nwb_production.NwbExportStatus.fetch())\n",
+    "print(\"Available NWB Export Statuses:\\n\")\n",
+    "print(status_df.to_string(index=False))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c4ae280d",
+   "metadata": {},
+   "source": [
+    "## 3. Explore NWB Export Job Table\n",
+    "\n",
+    "The `NwbExportJob` table stores information about each NWB export job, including the subject, session, and current status."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8fca4cf4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Display the NwbExportJob table structure\n",
+    "print(\"NwbExportJob Table Definition:\\n\")\n",
+    "print(nwb_production.NwbExportJob.describe())\n",
+    "print(\"\\n\" + \"=\"*80 + \"\\n\")\n",
+    "\n",
+    "# Check how many jobs exist\n",
+    "job_count = len(nwb_production.NwbExportJob())\n",
+    "print(f\"Total NWB export jobs in database: {job_count}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b01a57ec",
+   "metadata": {},
+   "source": [
+    "## 4. Explore Part Tables\n",
+    "\n",
+    "The NwbExportJob table has three part tables for different modalities:\n",
+    "- **BehaviorExport**: Tracks behavior data exports\n",
+    "- **EphysExport**: Tracks electrophysiology data exports  \n",
+    "- **ImagingExport**: Tracks imaging data exports"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dc4e351a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Display part table structures\n",
+    "print(\"BehaviorExport Table:\\n\")\n",
+    "print(nwb_production.NwbExportJob.BehaviorExport.describe())\n",
+    "print(\"\\n\" + \"=\"*80 + \"\\n\")\n",
+    "\n",
+    "print(\"EphysExport Table:\\n\")\n",
+    "print(nwb_production.NwbExportJob.EphysExport.describe())\n",
+    "print(\"\\n\" + \"=\"*80 + \"\\n\")\n",
+    "\n",
+    "print(\"ImagingExport Table:\\n\")\n",
+    "print(nwb_production.NwbExportJob.ImagingExport.describe())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "47959db0",
+   "metadata": {},
+   "source": [
+    "## 5. Explore Log and Validation Tables\n",
+    "\n",
+    "The schema also includes tables for logging status changes and storing validation results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fd6ea56a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# NwbExportLogStatus - tracks all status transitions\n",
+    "print(\"NwbExportLogStatus Table:\\n\")\n",
+    "print(nwb_production.NwbExportLogStatus.describe())\n",
+    "print(\"\\n\" + \"=\"*80 + \"\\n\")\n",
+    "\n",
+    "# NwbExportValidation - stores validation results\n",
+    "print(\"NwbExportValidation Table:\\n\")\n",
+    "print(nwb_production.NwbExportValidation.describe())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "497c21f7",
+   "metadata": {},
+   "source": [
+    "## 6. View ERD (Entity Relationship Diagram)\n",
+    "\n",
+    "Visualize the relationships between all tables in the NWB Production schema."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3195213a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Draw the Entity Relationship Diagram\n",
+    "dj.ERD(nwb_production)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0feb10e1",
+   "metadata": {},
+   "source": [
+    "## 7. Query Example: View Jobs by Status\n",
+    "\n",
+    "Let's see how to query jobs by their current status."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9ac3cd73",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Query jobs with specific statuses\n",
+    "# Status IDs: 0=QUEUED, 1=DATA_VALIDATION, 2=PROCESSING, 3=COMPLETED, -1=FAILED\n",
+    "\n",
+    "# Example: Get all active jobs (not completed or failed)\n",
+    "active_jobs = nwb_production.NwbExportJob & 'status_nwb_id >= 0 AND status_nwb_id < 3'\n",
+    "print(f\"Active jobs: {len(active_jobs)}\")\n",
+    "\n",
+    "# Example: Get all completed jobs\n",
+    "completed_jobs = nwb_production.NwbExportJob & 'status_nwb_id = 3'\n",
+    "print(f\"Completed jobs: {len(completed_jobs)}\")\n",
+    "\n",
+    "# Example: Get all failed jobs\n",
+    "failed_jobs = nwb_production.NwbExportJob & 'status_nwb_id = -1'\n",
+    "print(f\"Failed jobs: {len(failed_jobs)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b5ef6566",
+   "metadata": {},
+   "source": [
+    "## 8. Summary\n",
+    "\n",
+    "The NWB Production schema provides a complete framework for managing NWB export jobs:\n",
+    "\n",
+    "- **Job Tracking**: Submit jobs with modality selection (behavior, ephys, imaging)\n",
+    "- **Status Management**: Track jobs through pipeline stages (QUEUED → DATA_VALIDATION → PROCESSING → COMPLETED)\n",
+    "- **Audit Trail**: Complete history of status transitions with timestamps and error messages\n",
+    "- **Validation**: Store validation results with detailed metrics\n",
+    "\n",
+    "For more information, see:\n",
+    "- Web interface: Access through the Streamlit app's \"NWB Job Management\" page\n",
+    "- Background processor: `u19_pipeline/automatic_job/nwb_export_handler.py`\n",
+    "- Utilities: `u19_pipeline/nwb_production_utils.py`"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "u19-pipeline",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

scripts/migrate_nwb_production_schema.py

@@ -0,0 +1,36 @@
+"""
+Recreate u19_nwb_production tables via DataJoint.
+Run this after manually dropping the tables.
+"""
+import json
+import datajoint as dj
+
+# Bypass env-var overrides — use ct5868 credentials from config file
+with open("dj_local_conf.json") as _f:
+    _conf = json.load(_f)
+dj.config["database.host"]     = _conf["database.host"]
+dj.config["database.user"]     = _conf["database.user"]
+dj.config["database.password"] = _conf["database.password"]
+dj.config["database.port"]     = _conf["database.port"]
+dj.config["custom"]            = _conf["custom"]
+
+s    = _conf["custom"]["database.prefix"] + "nwb_production"
+conn = dj.conn(reset=True)
+print(f"Connected as: {dj.config['database.user']}")
+print(f"Schema:       {s}\n")
+
+# acquisition must be imported first so the `-> acquisition.Session` FK resolves
+from u19_pipeline import acquisition  # noqa: E402
+print("acquisition module loaded")
+
+print("Importing nwb_production (DataJoint will CREATE all tables)...")
+from u19_pipeline import nwb_production  # noqa: E402
+print("✓ Done\n")
+
+print("=== Tables now in schema ===")
+for r in conn.query(f"SHOW TABLES IN `{s}`"):
+    print(f"  {r[0]}")
+
+print("\n=== #nwb_export_status ===")
+for r in conn.query(f"SELECT * FROM `{s}`.`#nwb_export_status` ORDER BY status_id"):
+    print(" ", r)