Interstellar Notes 🌌🎶

Interstellar Notes is an end-to-end Automatic Music Transcription (AMT), score-to-audio alignment, and spectral visualization system.

It combines deep learning pitch estimation (GargantuaAMT), source separation (Demucs), score-to-performance alignment via Dynamic Time Warping (DTW), and an interactive web-based spectral visualizer.

---

✨ Features

• 🎹 GargantuaAMT Architecture: Deep CRNN model (CNN frontend + Bidirectional LSTM + Attention/Temperature sharpening) optimized for multi-track frequency resolution and note activation detection.
• ⏱️ Score-to-Audio DTW Alignment: Automated pipeline that aligns unconstrained audio recordings with MusicXML (.mxl) sheet music ground-truth targets.
• 🎛️ Audio Stem Separation: Integrated with Demucs for isolating instrument stems (bass, vocals, drums, other) prior to feature extraction.
• 🎼 Constant-Q Transform (CQT) Engine: High-resolution spectro-temporal feature extraction tuned to musical pitches and octaves.
• 🎨 Dual Visualization Suite:
  • Python Visualizer: High-fidelity overlays of CQT spectrograms and aligned pitch activation grids.
  • Spectral Player Web App: Interactive React + Vite frontend for inspecting spectral playback and transcription outputs.

---

📁 Repository Structure

interstellar_notes/
├── src/
│   ├── models/           # Deep learning models (GargantuaAMT architecture)
│   ├── audio_engine/     # Audio processing (CQT extraction & Demucs stem separation)
│   ├── transcriber/      # Data pipeline, dataset loaders, and inference engine
│   ├── trainer/          # Loss functions (BCE with logit sharpening) and training loops
│   └── utils/            # DTW alignment, MusicXML parsers, and plot visualizers
├── spectral_player/      # React + Vite web spectral player & visualizer
├── experiments/          # Audit scripts, synthetic dataset generators & experimental benchmarks
├── main.py               # Main pipeline entrypoint (Batch discovery & DTW alignment)
├── requirements.txt      # Python dependencies
└── LICENSE               # MIT License

---

🚀 Quickstart

1. Python Backend Setup

Clone the repository and install the dependencies:

git clone https://github.com/jzsalinas/interstellar_notes.git
cd interstellar_notes

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install requirements
pip install -r requirements.txt

2. Pre-trained Model Weights

Download the pre-trained weights (gargantua_final_gold.pth) from GitHub Releases and place them in the models/ directory:

mkdir -p models
# Place gargantua_final_gold.pth inside models/

3. Running the Data & DTW Alignment Pipeline

Place your audio files (.mp3) in data/raw/ and corresponding MusicXML files (.mxl) in data/labels/:

python main.py

The pipeline will discover audio-score pairs, compute CQT features, perform DTW alignment, and generate sanity-check visualizations in data/processed/aligned_checks/.

4. Web Spectral Player

To launch the web visualization player:

cd spectral_player
npm install
npm run dev

Open http://localhost:5173 in your browser to interact with the spectral player.

---

🔬 Model Architecture: GargantuaAMT

GargantuaAMT is designed specifically for multitrack pitch identification:

1. CNN Frontend: Multi-channel 2D convolutional layers taking multi-stem CQT inputs and extracting pitch-invariant spatial frequency maps.
2. Recurrent Backbone: Bidirectional LSTM layers capturing temporal note sustain and onset continuity across time frames.
3. Logit Bias Initialization: Pre-conditioned output layer configured for sparse musical event detection (favouring silence until clear spectral energy is present).

---

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🤝 Contributing

Contributions, issues, and feature requests are welcome! Feel free to check the issues page or submit a Pull Request.