ip26A is an AI-powered system that translates movie and video dialogue between languages while preserving the original actor’s voice and keeping background music and sound effects untouched.
The MVP focuses on English, Korean, and Japanese, enabling high-quality localization without traditional dubbing.
- 🎙️ Actor voice cloning for translated dialogue
- 🌍 Multilingual translation (EN ↔ KR ↔ JP)
- 🎶 Background audio preservation (music & effects)
- 🎬 Movie & episodic video support
- ⚡ Modular, scalable Python architecture
- Upload a video file
- Extract and separate dialogue from background audio
- Transcribe dialogue (ASR)
- Translate dialogue text
- Re-synthesize speech using the actor’s cloned voice
- Re-mix translated dialogue with original background audio
- Output a fully localized video
- Python – Core orchestration & AI pipeline
- PyTorch – Model inference
- FastAPI – API layer
- FFmpeg – Audio/video processing
ip26A/
├── app/
│ ├── main.py
│ ├── audio_processing/
│ ├── speech_recognition/
│ ├── translation/
│ ├── voice_cloning/
│ ├── video/
│ └── utils/
├── models/
├── data/
├── tests/
└── docs/
✅ English, Korean, Japanese translation ✅ Voice-preserved dubbing ✅ Background audio intact
🚫 Real-time translation (future) 🚫 Public voice uploads (enterprise-only)
- Movie studios
- Streaming platforms
- Anime distributors
- Localization companies
ip26A is built for licensed and consent-based voice usage only. Actor voice models must be legally authorized.
See milestones and timeline below.
[ Video File ]
│
▼
[ Audio Extractor ] ──► [ Background Audio Track ]
│
▼
[ Dialogue Separator ]
│
▼
[ Speech-to-Text (ASR) ]
│
▼
[ Translation Engine ]
│
▼
[ Voice Cloning TTS ]
│
▼
[ Dialogue Mixer ] ◄─── [ Background Audio Track ]
│
▼
[ Final Video Output ]
- Audio Extractor: Pulls raw audio from video
- Dialogue Separator: Isolates speech from music/effects
- ASR: Converts speech to text
- Translator: Converts text between languages
- Voice Cloning TTS: Recreates dialogue in actor’s voice
- Mixer: Combines translated speech with original background audio
- Project setup & repository structure
- FFmpeg video/audio extraction
- Basic API skeleton (FastAPI)
Output: Video → clean dialogue & background tracks
- Integrate ASR for EN / KR / JP
- Language detection
- Timestamped dialogue segments
Output: Accurate transcripts per scene
- Text translation between EN / KR / JP
- Sentence alignment & timing preservation
Output: Translated dialogue text
- Actor voice embedding system
- TTS synthesis using cloned voices
- Emotion & pacing alignment
Output: Natural-sounding translated speech
- Mix translated dialogue with original background
- Sync lip timing as closely as possible
Output: Watchable translated scenes
- End-to-end pipeline testing
- Short movie/episode demo
- Documentation & demo assets
Output: MVP-ready product for demos & pitches
- More languages
- Studio dashboard
- Batch processing
- API monetization
- Streaming platform integrations
This section lists practical, production-proven models suitable for an MVP that can later scale to studio-grade quality.
Purpose: Isolate spoken dialogue while preserving music and sound effects.
- Demucs v4 (Hybrid Transformer)
Why:
- Industry-grade music/voice separation
- Strong performance on movie soundtracks
- Actively maintained
Output:
dialogue.wavbackground.wav
Purpose: Convert dialogue audio into accurate text with timestamps.
- OpenAI Whisper (large-v3)
Languages Supported:
- English
- Korean
- Japanese
Why:
- High accuracy on accented & emotional speech
- Handles movie-quality audio well
- Timestamp support for lip-sync alignment
Purpose: Translate dialogue text between languages.
- NLLB-200 (Meta)
Why:
- Strong Asian language support
- Consistent sentence structure
- Open-source and scalable
- MarianMT (Helsinki-NLP)
Purpose: Recreate translated dialogue using the original actor’s voice.
- OpenVoice v2 – Voice cloning
- XTTS v2 (Coqui) – Multilingual TTS
Why:
- Supports Korean, Japanese, English
- Zero-shot or few-shot voice cloning
- Emotion and tone preservation
Purpose: Sync translated speech with original timing and merge with background.
- Montreal Forced Aligner (optional)
- FFmpeg for final mixing
This section is critical for studios and distributors.
Problem: Translated speech duration differs from original dialogue.
Mitigation:
- Sentence-level timestamp alignment
- Speed-controlled TTS generation
- Phoneme-aware trimming
Problem: Actor voice sounds artificial or inconsistent.
Mitigation:
- Scene-specific voice embeddings
- Emotion tagging (angry, calm, whisper)
- Quality threshold validation
Problem: Cultural nuance lost in translation.
Mitigation:
- Context-aware sentence grouping
- Human-in-the-loop review option
- Studio custom glossaries
Problem: Music leaks into dialogue channel.
Mitigation:
- Post-separation spectral cleanup
- Multi-pass separation
- Manual override fallback
Problem: Unauthorized voice usage.
Mitigation:
- Contract-based voice enrollment
- Watermarked voice models
- Studio-only deployment
ip26A is designed for B2B and enterprise licensing, not consumer use.
- ₦2 – ₦5 per finished minute (MVP)
- Discounts for bulk content
-
Monthly or yearly license
-
Includes:
- Actor voice storage
- Batch processing
- Priority compute
- Custom pricing
- On-prem or private cloud deployment
- 60–80% cost reduction vs manual dubbing
- Faster global releases
- Consistent actor identity
- No re-recording logistics
ip26A presents a modular AI-driven framework for multilingual movie dialogue translation with preserved speaker identity. The system integrates speech separation, neural machine translation, and multilingual voice cloning to deliver studio-quality localization while maintaining original audio fidelity.
Global film distribution faces increasing demand for rapid, high-quality localization. Traditional dubbing pipelines are costly, time-consuming, and often compromise emotional authenticity. ip26A proposes an AI-based alternative that maintains actor identity across languages.
The ip26A pipeline consists of six primary modules:
- Dialogue separation
- Automatic speech recognition
- Neural machine translation
- Voice embedding extraction
- Multilingual speech synthesis
- Audio recomposition
Each module operates independently, allowing flexible upgrades.
Hybrid transformer-based source separation models are employed to isolate dialogue from complex cinematic audio mixtures.
Multilingual ASR models transcribe dialogue with timestamp precision, enabling temporal alignment in synthesis.
Neural translation models optimized for East Asian and Indo-European language pairs perform context-aware translation.
Speaker embeddings are extracted from licensed reference audio and reused to synthesize translated dialogue while preserving vocal identity.
ip26A enforces consent-based voice modeling, contractual authorization, and watermarking to prevent misuse.
The proposed system significantly reduces localization cost and time while increasing audience immersion, making it suitable for global streaming platforms.
ip26A demonstrates that AI-assisted localization can preserve artistic integrity while enabling scalable global distribution.