OpenBMB/VoxCPM

VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning

What it solves

VoxCPM is a tokenizer-free Text-to-Speech (TTS) system designed to generate highly natural, expressive, and studio-quality speech. It addresses the limitations of discrete tokenization in traditional TTS, allowing for more fluid and realistic multilingual speech generation, creative voice design, and high-fidelity voice cloning.

How it works

The system uses an end-to-end diffusion autoregressive architecture built on a MiniCPM-4 backbone. It directly generates continuous speech representations instead of discrete tokens. VoxCPM2, the latest version, is a 2B parameter model trained on over 2 million hours of multilingual speech data. It utilizes an asymmetric encode/decode design via AudioVAE V2 to output 48kHz audio directly from 16kHz reference audio, incorporating built-in super-resolution.

Who it’s for

  • Content Creators: Those needing high-quality, multilingual speech synthesis for videos or podcasts.
  • Developers: Those building AI voice applications requiring real-time streaming, voice cloning, or custom voice design.
  • Enterprise: Companies looking for commercial-ready, open-source TTS solutions (Apache-2.0 license).

Highlights

  • 30-Language Support: Synthesizes speech in 30 different languages and various Chinese dialects without needing language tags.
  • Voice Design: Creates new voices based on natural-language descriptions (e.g., age, gender, tone) without reference audio.
  • Controllable Voice Cloning: Clones timbre from a short clip while allowing style guidance for emotion and pace.
  • Ultimate Cloning: Reproduces every vocal nuance, including rhythm and style, by using both reference audio and its transcript.
  • Studio-Quality Audio: Directly outputs 48kHz audio with built-in super-resolution.
  • High Performance: Supports real-time streaming with RTF as low as 0.13 on an NVIDIA RTX 4090 when accelerated by Nano-vLLM or vLLM-Omni.
  • Deployment Flexibility: Available via Python API, CLI, and high-performance C++ inference engines like llama.cpp-omni for on-device use.

Related

  • Project
    wildminder/ComfyUI-VoxCPM

    A ComfyUI custom node integration for VoxCPM, a tokenizer-free TTS system that enables expressive speech generation, natural language voice design, and advanced voice cloning.

  • Project
    netease-youdao/Confucius4-TTS

    Confucius4-TTS is an LLM-based text-to-speech engine that enables zero-shot, cross-lingual voice cloning across 14 languages without requiring reference transcripts.

  • Project
    Saganaki22/ComfyUI-OmniVoice-TTS

    ComfyUI nodes for OmniVoice, providing zero-shot multilingual text-to-speech with voice cloning, voice design, and multi-speaker dialogue support for over 600 languages.

  • Project
    tronghieuit/v-tts

    A lightweight Vietnamese text-to-speech system featuring multi-speaker synthesis and zero-shot voice cloning that runs efficiently on CPU.

  • Project
    QwenAudio/CosyVoice

    CosyVoice is an advanced, LLM-based text-to-speech system designed for high-quality, zero-shot multilingual speech synthesis and voice cloning.