facebookresearch/seamless_communication

Foundational Models for State-of-the-Art Speech and Text Translation

What it solves

Seamless is a family of AI models designed to make communication across different languages more natural and authentic. It addresses the challenges of translating speech and text across nearly 100 languages while maintaining the original speaker's voice style, prosody (such as speech rate and pauses), and the ability to perform translations in real-time.

How it works

The project consists of several specialized models that can be combined or used independently:

  • SeamlessM4T: A foundational multimodal machine translation model that handles speech-to-speech, speech-to-text, text-to-speech, and text-to-text translation, as well as automatic speech recognition (ASR).
  • SeamlessExpressive: A model focused on preserving prosody and voice style during speech-to-speech translation.
  • SeamlessStreaming: A model that enables simultaneous translation and streaming ASR.
  • Seamless: A unified model that combines the capabilities of streaming and expressivity.

These models are powered by the fairseq2 library and utilize the W2v-BERT 2.0 speech encoder. For deployment on resource-constrained platforms, unity.cpp allows the models to run using the GGML C tensor library.

Who it’s for

Researchers and developers building multilingual translation applications, real-time communication tools, and systems that require high-fidelity voice preservation across languages.

Highlights

  • Massive Multilinguality: Supports nearly 100 languages across multiple modalities.
  • All-in-One Translation: Handles S2ST, S2TT, T2ST, T2TT, and ASR in a single framework.
  • Prosody Preservation: Captures speech rate and pauses to maintain the authentic feel of the original speaker.
  • Real-time Capability: Supports simultaneous streaming translation.
  • Efficient Inference: Includes a unity.cpp implementation for easier integration on various platforms.

Related

  • Project
    OpenMOSS/MOSS-TTS

    A family of open-source speech and sound generation models providing high-fidelity TTS, expressive dialogue, voice design, real-time interaction, and sound effect generation.

  • Project
    fishaudio/fish-speech

    A state-of-the-art multilingual text-to-speech system that uses a Dual-AR architecture to provide realistic voice cloning and fine-grained emotional control via natural language tags.

  • Dispatch
    Hugging Face and Cerebras Real-Time Voice AI with Gemma 4

    Hugging Face and Cerebras have developed an open, cascaded speech-to-speech pipeline using Gemma 4 31B to enable natural, low-latency voice AI interactions.

  • Project
    2noise/ChatTTS

    ChatTTS is a generative text-to-speech model optimized for conversational dialogue, providing natural, expressive speech with control over laughter and pauses.

  • Project
    resemble-ai/chatterbox

    A family of open-source text-to-speech models providing low-latency voice cloning and multilingual speech generation across various model sizes for GPU and CPU deployment.