IQ Data Stream Compression: PCA vs. Autoencoder

Hackathon project (hosted by Rohde & Schwarz) comparing classical and learned compression for high-rate IQ data streams from test & measurement equipment (oscilloscopes, signal generators).

---

Problem

Modern T&M devices generate massive IQ (in-phase/quadrature) data streams that are expensive to store and transmit, especially for remote/edge processing. The goal: compress the stream while keeping it usable for downstream signal processing — ideally lossless or near-lossless.

Two approaches were built and compared:

1. PCA — classical linear dimensionality reduction, fast and cheap but limited to linear structure.
2. Autoencoder — a TensorFlow/Keras neural network with a bottleneck layer, trained to minimize MSE reconstruction loss; captures nonlinear structure PCA can't.

Results

Autoencoder over sliding windows of the raw IQ stream:

Metric	Value
Compression ratio	16.0 : 1
EVM (reconstruction error)	0.1182 (11.82%) / −18.55 dB
Final test loss (MSE)	0.0030

Takeaways: PCA is a strong, near-free baseline for linear signal structure. The autoencoder captures more of the nonlinear structure in the modulated IQ stream at a fixed compression ratio, at the cost of needing training data and GPU time. A hybrid — PCA as a cheap first pass, autoencoder for the residual — looked like the most promising direction for further work.

Files

├── src/
│   ├── autoencoder_baseline.py    # First working autoencoder (single file, fixed hyperparameters)
│   ├── autoencoder_optuna.py      # Later iteration: GPU support + Optuna hyperparameter search
│   ├── comparison.py              # PCA vs. autoencoder evaluation (EVM, compression ratio)
│   ├── sliding_window.py          # Windowing generator for streaming IQ data into the model
│   ├── dwt.py                     # Discrete wavelet transform compression baseline
│   ├── dataconversion.py          # .npz -> .csv conversion utility
│   ├── load.py                    # Quick-look plotting of a raw capture
│   └── optuna_study_analysis.py   # Visualizes Optuna hyperparameter search results
├── data/                          # Sample IQ captures (8PSK, pi/4-DQPSK; various symbol/sample rates)
└── figures/                       # Result plots (constellation, EVM, loss curves)

autoencoder_baseline.py and autoencoder_optuna.py are two stages of the same model — the first is the minimal working version, the second adds GPU acceleration and automated hyperparameter search (Optuna) once the basic approach was validated.

Quick Start

pip install numpy pandas tensorflow scikit-learn scipy pywavelets matplotlib optuna
python src/autoencoder_baseline.py

Fundamentals

• PCA: eigendecomposition of the covariance matrix, keep the top-k components. Efficient, but assumes linear structure.
• Autoencoder: encoder/decoder network learns a nonlinear compressed representation directly from data, at the cost of requiring training and more compute.

Future Directions

• Variational or Transformer-based compression architectures
• Validation against real-world (non-synthetic) capture data
• Edge deployment for real-time, low-power compression
• Adaptive compression that responds to channel conditions