SWivid/F5-TTS

Official code for "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"

What it solves

F5-TTS addresses the challenge of creating fluent and faithful text-to-speech (TTS) generation. It aims to provide a system that can generate high-quality speech that sounds natural and remains faithful to the input text, while improving the speed of both training and inference compared to previous models.

How it works

F5-TTS utilizes a Diffusion Transformer architecture combined with ConvNeXt V2. It employs a technique called flow matching to generate speech. The project also includes "Sway Sampling," an inference-time flow step sampling strategy designed to boost performance. For those seeking a closer reproduction of the E2 TTS paper, the repository also provides an E2 TTS implementation using a Flat-UNet Transformer.

Who it’s for

This tool is for developers and researchers working on speech synthesis, as well as users who want to generate multi-style or multi-speaker audio from text. It is suitable for those who need a high-performance TTS engine capable of running on various hardware, including NVIDIA, AMD, Intel GPUs, and Apple Silicon.

Highlights

  • High Performance: Optimized for faster training and inference using Diffusion Transformer and ConvNeXt V2.
  • Flexible Deployment: Supports multiple hardware backends and provides a Docker image for easy setup.
  • Advanced Sampling: Includes Sway Sampling to improve inference-time performance.
  • Versatile Interface: Offers both a Gradio web interface (supporting voice chat and multi-speaker generation) and a CLI for automated tasks.
  • Enterprise Ready: Includes deployment solutions using Triton and TensorRT-LLM for low-latency production use.

Related

  • Project
    fishaudio/fish-diffusion

    A diffusion-based training framework for Text-to-Speech, Singing Voice Synthesis, and Singing Voice Conversion that supports multi-speaker training and efficient hardware utilization.

  • Project
    shivammehta25/Matcha-TTS

    Matcha-TTS is a fast, non-autoregressive text-to-speech architecture that uses conditional flow matching to enable high-quality, efficient speech synthesis.

  • Project
    KoljaB/RealtimeTTS

    A Python library for low-latency text-to-speech that converts strings and LLM token streams into audio using a wide variety of local and cloud engines.

  • Project
    neuphonic/neutts

    A collection of open-source, on-device text-to-speech models that enable natural-sounding speech and instant voice cloning locally on mobile and embedded devices.

  • Project
    Aratako/Irodori-TTS

    A Flow Matching-based text-to-speech model that supports zero-shot voice cloning and multi-modal style control using text, reference audio, and captions.