OpenMOSS/MOVA
MOVA: Towards Scalable and Synchronized Video–Audio Generation
What it solves
MOVA is a foundation model designed to generate high-fidelity video and synchronized audio simultaneously. It eliminates the need for cascaded pipelines where audio is added as a separate, often misaligned step, solving the problem of poor lip-sync and unsynchronized sound effects in open-source video generation.
How it works
MOVA uses an asymmetric dual-tower architecture that combines pre-trained video and audio towers. These towers are fused via a bidirectional cross-attention mechanism, allowing the model to synthesize both modalities in a single inference pass. It supports both Text-to-Video-Audio (T2VA) and Image-to-Video-Audio (I2VA) workflows.
Who it’s for
This project is for AI researchers, developers, and content creators who need high-quality, synchronized audiovisual content. It is particularly useful for those wanting to implement multilingual lip-synchronization or environment-aware sound effects.
Highlights
- Native Bimodal Generation: Generates video and audio in one pass to prevent error accumulation and ensure perfect alignment.
- Precise Lip-Sync: Achieves state-of-the-art performance in multilingual lip-synchronization.
- Open-Source Ecosystem: Provides model weights, inference code, training pipelines, and LoRA fine-tuning scripts.
- Broad Integration: Supports SGLang for high-throughput inference, ComfyUI via a community plugin, and provides a hosted API.
- Flexible Hardware Support: Includes optimizations for VRAM offloading and support for Ascend NPUs.