🎵 Deshimmer: AI-Audio Artifact Reconstruction, Denoiser & Intelligent Optimizer

Deshimmer is a highly advanced, clinical-grade audio reconstruction suite designed to diagnose, isolate, and surgically suppress the distinct high-frequency artifacts, stationary "whistles", and phase incoherence ("swish") generated by state-of-the-art AI music platforms (such as Suno AI, Udio, and other neural audio decoders).

Beyond simple static filtering, Deshimmer embeds a stateful, multi-stage DSP engine alongside a world-first multi-objective Bayesian optimization framework (powered by Optuna) to automatically analyze your audio and find the mathematically optimal parameter set for artifact suppression while preserving pristine musicality.

---

📖 Table of Contents

1. 🎯 The AI-Audio Artifact Problem
2. ✨ Key Features
3. ⚙️ DSP Engine Architecture Deep-Dive
4. 🧠 The Optuna Auto-Tuning System
5. 🖥️ Interactive UI (Gradio App)
6. 🔌 C++ VST3 / AU Plugin (DeAI)
7. 🚀 Installation & Quick Start
8. 📊 CLI and Visual Forensics Suite
9. 🎛️ Parameters Reference Glossary
10. 🔬 How the Core DSP Works
11. 🤝 Contributing & License

---

🎯 The AI-Audio Artifact Problem

Generative diffusion models and neural vocoders (like GANs and Flow-Matching architectures) suffer from systematic reconstruction errors. When rendering high frequencies, they produce:

• "Birdies" & Shimmer: High-Q, flickering spectral outliers (typically in the 5.1 – 7.2 kHz or 5 – 14 kHz bands) that oscillate rapidly, creating an unnatural, metallic, "crystalline" glaze.
• Stationary Whines: Pure-tone ringing resonances caused by grid-line sampling or generator clock-harmonics.
• Neural Decoder "Swish": Moving phase incoherences and decorrelation errors in the high frequencies, turning natural transients and noise-like elements into watery, "phasey" swish.

Deshimmer addresses these artifacts surgically, separating them from legit broadband musical content.

---

✨ Key Features

• 🎚️ Stateful Multi-Stage DSP: Sequential processing pipeline including:
  • Surgical Shimmer Suppression: Dynamic thresholding with adaptive baseline tracking.
  • Smart Spectral Denoiser: Minimum-statistics tracking to manage noise floor without gating artifacts.
  • Intelligent De-Resonator (Dynamic EQ): Identifies stationary resonances with temporal persistence modeling.
  • Phase-Swish Repair: Adaptive inter-frame and inter-bin phase coherence reconstruction.
• 🧠 Bayesian Hyperparameter Optimizer: Two-layer auto-tune (Cheap deterministic analysis + staged multi-objective Optuna search) that solves for the optimal DSP parameters.
• 🖥️ Full-Featured Gradio Web UI: Visual real-time spectrograms, modulation spectrum plots, looping preview sections with crossfading, pre-baked presets, and instant download triggers.
• 🔌 Native C++ DAW Plugin (DeAI): Build and run low-latency realtime deshimmering directly in Reaper, Logic, Ableton, or Pro Tools via a JUCE-based C++ engine.
• 🔬 Diagnostic Forensics: Analyze tracks, generate artifact-only WAV outputs, plot residual heatmaps, and run full-reference synthetic benchmarks.

---

⚙️ DSP Engine Architecture Deep-Dive

Deshimmer's pipeline is constructed out of decoupled, highly optimized processing blocks in master.py (accelerated via Numba JIT vectorization):

                       ┌─────────────────────────┐
                       │    Input Audio (L/R)    │
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │     STFT Analysis       │
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │  Smart Spectral Denoise │ <── Min-Statistics Tracker
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │  Adaptive De-Resonator  │ <── Persistence-Weighted EQ
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │    Shimmer Suppressor   │ <── Flatness & Transient Gate
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │   Swish Phase Repair    │ <── Coherence Smoothing
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │   Stereo Decorrelation  │ <── Phase-Lock Breaker
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │   Nuclear Resynthesis   │ <── Sub-band HF Exciter
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │    iSTFT Synthesis      │
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │ Lookahead Peak Limiter  │
                       └────────────┬────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │   Processed Audio       │
                       └─────────────────────────┘

1. Smart Spectral Denoiser (Taste-Neutral)

Targets broadband high-frequency hiss and neural vocoder leakage.

• Minimum-Statistics Tracking: Tracks the absolute noise floor on a per-bin basis within a rolling window (dn_minwin_ms), preventing "gating" artifacts.
• Rise Speed Constraints: Limits how fast the noise floor estimate can rise (dn_up_db_per_s) so that musical tails are never cut off.
• Frequency Smoothing: Smooths attenuation gains across neighboring bins (dn_freq_smooth_bins) to eliminate watery "musical noise."

2. Adaptive De-Resonator (Dynamic EQ)

Identifies and suppresses stationary ringing or constant whistles.

• Persistence Modeling: Computes a time-weighted persistence index (deq_persist_ms) to distinguish constant whistles (like clock noise) from fast-moving musical harmonics.
• Tonal Boosting: Dynamically scales threshold scaling when frames are highly tonal to prevent detuning instruments.
• Magnitude Inpainting: Caps outlier spikes back down to the local median rather than brutally cutting gain.

3. Shimmer Suppressor (Core)

Suppresses flickering, high-frequency "birdies."

• Spectral Flatness Gating: Analyzes the geo/arith mean of the power spectrum; limits processing to noise-like frames while bypassing tonal/musical content.
• Energy Flux Transient Gate: Tracks spectral energy jumps; instantly bypasses processing during rapid attacks (drum hits, plucks) to preserve crisp transients.
• Density Protection: Computes the fraction of out-of-bounds bins. If many bins spike, it is classified as a broadband musical event (e.g. vocal belt, brass stab) and bypassed.

4. Swish Phase Repair & Stereo Decorrelation

Restores spatial depth and removes watery "swish."

• Phase Coherence Smoothing: Employs adaptive inter-frame and inter-bin phase coherence reconstruction to glue together fractured phase alignments.
• HF Stereo Decorrelation: Breaks unnatural, synthetic L/R phase locking in upper frequencies, widening the stereo image and injecting organic acoustic space.

5. "Nuclear" High-Frequency Resynthesis

When AI artifacts are completely inseparable from the musical content, activate Nuclear Mode:

• Complete Upper Band Wipe: Shaves off all high frequencies above a configurable threshold.
• Harmonic Excitation: Drives lower mid-range frequencies through a tanh wave-shaper to excite high-order harmonics.
• Transient Alignment: Aligns the timing of generated harmonics with low-band onsets.
• Confidence-Blended Mix: Blends the resynthesized high end back into the track only in regions where artifact confidence is high.

---

🧠 The Optuna Auto-Tuning System (World First)

Finding the correct knobs to deshimmer a track manually can be challenging. Deshimmer implements an automated, two-layer optimization framework in auto_tune.py that discovers the perfect parameters for you.

Layer 1: Deterministic Signal Analyzer (analyze)

A highly efficient, deterministic pass that scans the entire audio file to extract priors:

• Median-Residual Band Scan: Slides a search window across high frequencies to find the exact frequency boundaries and dB-strength of the shimmer band.
• Broadband Floor & Dynamics: Measures the overall noise floor and dynamic range.
• Persistent Peak Detection: Pinpoints exact resonance frequencies.
• Region Sampler: Picks representative chunks (Quiet, Loud, Transient-rich, Vocal, Tail/Sustain) to use for the next layer.

Layer 2: Staged Multi-Objective Bayesian Optimizer (refine)

A state-of-the-art hyperparameter search using Optuna's NSGA-II (Non-dominated Sorting Genetic Algorithm II) sampler. Optimization is done sequentially across stages to isolate variables: $$\text{Stage A (Shimmer)} \rightarrow \text{Stage B (Denoise)} \rightarrow \text{Stage C (Deres)} \rightarrow \text{Stage D (Swish)}$$

At each stage, it optimizes 5 competing objectives concurrently:

1. Artifact Reduction (Maximize attenuation of outliers)
2. Musical Content Damage (Minimize changes to out-of-band energy)
3. Difference Signal Leakage (Minimize transient/harmonic leakage into the removed "diff" signal)
4. Stereo Image Damage (Minimize deviation in stereo width delta)
5. Spectral Tilt & Loudness Loss (Minimize overall volume and tonal balance shifts)

Optimization Profiles:

• Safe / Conservative: Prioritizes preserving the original musical content and avoiding leakage above all else.
• Aggressive: Prioritizes maximal artifact removal and suppression.
• Balanced (Default): Leverages Pareto-Front Analysis to mathematically select the sweet spot on the multi-dimensional Pareto frontier.

---

🖥️ Interactive UI (Gradio App)

The Gradio application (ui_gradio.py) provides a world-class studio workspace.

python ui_gradio.py

Visual Forensics Panels

• Spectrograms: Real-time side-by-side comparison of Input, Processed, and Difference (Removed Artifacts).
• Modulation Spectrum: Tracks the temporal "flicker" rate of high frequencies, visually proving when birdies have been stabilized.
• Attenuation Map: Draws an explicit heatmap showing exactly which bins, frames, and stages were attenuated.

Studio Control Features

• Region Selector & Looping Preview: Select a short preview area, turn on the looped crossfader, and hear real-time A/B comparisons with instantaneous parameter updates.
• Dedicated Auto-Tune Panel:
  • Run Analyze with one click to set custom DSP defaults.
  • Run Refine to kick off the Optuna optimizer; watch the real-time progress bar, diagnostic logs, and a markdown report describing the optimization results.
• Pre-baked Presets: Standard, Aggressive, Phase Repair, Denoise Only, or Nuclear Resynthesis.
• Full Renderer: Process the full track with high-fidelity padding and download the WAV immediately.

---

🔌 C++ VST3 / AU Plugin (DeAI)

Located in vst/DeAI, this is a fully compiled C++ audio plugin built on top of the JUCE Framework, allowing real-time, zero-latency artifact cleanup inside your digital audio workstation (DAW).

Capabilities

• Translates the core Python logic into high-performance, vectorized C++ loop constructs.
• Dynamic median-residual baseline scans.
• Real-time transient masking and spectral flatness filters.
• Low-latency persistence-weighted whistle tracking.

Compilation (MacOS Example)

1. Install JUCE.
2. Open vst/DeAI/DeAI.jucer inside Projucer.
3. Export to Xcode (or Make/VS on Windows/Linux).
4. Build target DeAI - VST3 or DeAI - AU.

---

🚀 Installation & Quick Start

1. Clone the repository

git clone https://github.com/TheApeMachine/deshimmer.git
cd deshimmer

2. Install dependencies

# Core DSP & CLI
pip install -r requirements.txt

# (Optional) Gradio Web UI
pip install -r requirements-ui.txt

# (Optional) Real-time playback
pip install -r requirements-rt.txt

3. Process a file immediately via CLI

python deshimmer.py input.wav output.wav

---

📊 CLI and Visual Forensics Suite

Dynamic Probe (birdie_probe.py)

Isolate and inspect suspected artifact regions:

python birdie_probe.py input.wav --outdir probe_output --t0 23.0 --dur 4.0

Generates:

• artifact.wav (the isolated noise outlier component)
• roi_band_spectrogram.png (magnified spectrogram of the target band)
• roi_residual_map.png (high-contrast heatmap of outliers above baseline)

Hot-Reloading Real-time Player (realtime_player.py)

Play the full song and tune parameters in your favorite text editor:

python realtime_player.py input.wav --params realtime_params.json

Hot-reloads settings instantly on saving realtime_params.json and performs a clean crossfade between parameters.

Synthetic Benchmarking (synthetic_benchmark.py)

Inject artificial, drift-prone AI artifact families into pristine audio references to test suppression quality:

python synthetic_benchmark.py

Tracks full-reference metric scores (Artifact reduction vs. Music leakage) to benchmark DSP engine updates.

---

🎛️ Parameters Reference Glossary

Parameter Group	Knobs	Default	Description
Frequency Band	start_hz / end_hz center_hz / width_cents edge_hz	5100.0 / 7200.0 - / - 200.0	Targets the boundary frequencies of the shimmer band. Alternatively specify using musical cents. edge_hz smooths out the band edges.
Shimmer Suppressor	thr_db slope density_lo / density_hi flat_start / flat_end	8.0 0.6 0.02 / 0.15 0.25 / 0.70	Outlier threshold above median. Slope defines knee aggression. Density thresholds protect broadband events. Flatness thresholds gate noise-like frames.
Spectral Denoiser	denoise dn_floor_db dn_minwin_ms dn_freq_smooth_bins	0.0 -18.0 400.0 3	Denoise wet blend. Gain floor (dB) protects low-level acoustics. Window size for tracking minima. Bins for frequency-wise smoothing.
De-Resonator (EQ)	deres deq_thr_db deq_persist_ms deq_persist_thr_db	0.0 6.0 600.0 2.5	De-resonator wet blend. Tonal peak detection threshold. Length of rolling window to verify static resonance persistence.
Swish Phase Repair	swish_repair swish_time_amt swish_freq_amt	0.0 0.55 0.30	Wet blend for phase repair. Extent of phase smoothing across time (frames) and frequency (bins).
Advanced Controls	ms_process noise_resynth hf_decorrelate	False 0.0 0.0	Mid/Side processing. Noise phase resynthesis level. High-frequency L/R decorrelation amount.
Nuclear Resynth	hf_resynth hf_lp_hz hf_drive	False 3000.0 2.0	Enables complete HF band replacement. Cutoff for low pass. Saturation level for harmonic generation.

---

🔬 How the Core DSP Works

1. Short-Time Fourier Transform (STFT): Decomposes the signal into an overlapping matrix of complex-valued frequency bins over time.
2. Spectral Profiling: Under the hood, a wide-median baseline profile is computed. Positive-going deviations (peaks) and negative-going deviations (valleys) are isolated.
3. Multi-Stage Masking:
   • Transient & Density Check: High energy jumps and wide-scale frequency clusters are mapped; if active, suppression gain masks are attenuated towards unity (bypassed).
   • Persistence Verification: Bins indicating constant resonance over time are weighted, feeding the De-resonator.
   • Noise Gating: Bins matching the spectral flatness profile of pure noise undergo soft-knee suppression or magnitude inpainting.
4. Phase Coherence Smoothing: Complex phasors are smoothed using inter-frame and inter-bin circular-mean estimators.
5. Inverse STFT (iSTFT): Reconstructs the time-domain signal using overlap-add windowing with full boundary padding compensation to prevent clicks.
6. Mastering Limiter: Final signal path enters a stereo lookahead brickwall limiter to catch any clipping on output.

---

🤝 Contributing & License

Deshimmer is open-source and released under the MIT License. If you discover highly successful parameter recipes for specific styles of AI audio models, please share them or open a pull request!

---

⚠️ Disclaimer

Deshimmer is highly effective at reducing high-frequency artifacts, but it is not a magic wand. Highly corrupted low-resolution AI tracks might require a blend of deshimmering, noise-reduction, and master compression to achieve standard audio profiles. Always listen critically.

Made with passion, mathematical frustration at AI artifacts, and Python.