jasonppy/VoiceCraft

What it solves

VoiceCraft addresses the difficulty of performing high-quality, zero-shot speech editing and text-to-speech (TTS) using "in-the-wild" data (such as podcasts, audiobooks, and internet videos). It allows users to clone an unseen voice using only a few seconds of reference audio, making it capable of both generating new speech from text and modifying existing audio recordings.

How it works

VoiceCraft is a token infilling neural codec language model. It treats speech as a sequence of tokens (using Encodec) and uses a language model approach to "fill in" the missing parts of the audio tokens based on thetext transcript and a reference audio prompt. This architecture allows it to handle both TTS (generating speech from scratch) and speech editing (replacing or modifying specific parts of an existing audio clip).

Who it’s for

• Audio Engineers and Content Creators: For editing speech audio without needing the original speaker to re-record.
• AI Researchers: Those interested in neural codec language models and zero-shot voice cloning.
• Developers: Users who want to integrate high-quality TTS and speech editing into their applications via standalone scripts or Docker.

Highlights

• Zero-Shot Voice Cloning: Clones unseen voices with just a few seconds of reference audio.
• Dual Functionality: Supports both text-to-speech generation and speech editing.
• In-the-Wild Performance: Trained on diverse, real-world audio data including audiobooks and podcasts.
• Flexible Deployment: Provides multiple ways to run inference, including Google Colab, Docker, and standalone Python scripts.