Stability-AI/stable-codec

A family of state-of-the-art Transformer-based audio codecs for low-bitrate high-quality audio coding.

What it solves

It provides a high-quality speech coding system that compresses audio into low-bitrate tokens while maintaining high reconstruction quality. It specifically addresses the need for efficient audio representation that is useful for downstream tasks like text-to-speech (TTS).

How it works

The system uses a Transformer-based architecture with sliding window attention to encode audio into latents and tokens. It employs a Finite Scalar Quantization (FSQ) bottleneck to discretize the audio. To improve the usability of tokens for downstream AI tasks, the model can be finetuned using a Connectionist Temporal Classification (CTC) head to regress force-aligned phoneme tags from the latents.

Who it’s for

Researchers and developers working on speech synthesis, audio compression, and generative audio models who need an efficient way to represent speech as discrete tokens.

Highlights

  • Flexible Bitrate Control: Supports post-hoc bottleneck configuration to adjust the token dictionary size and bits per second (bps) after training.
  • Optimized for Downstream Tasks: The stable-codec-speech-16k variant is specifically finetuned to boost latent semantics for better performance in TTS applications.
  • High-Efficiency Inference: Utilizes FlashAttention for optimized processing of sliding window attention.
  • Comprehensive Training Tools: Includes scripts for finetuning and support for CTC loss to integrate phoneme-level alignment.

Related

  • Project
    Xiaobin-Rong/gtcrn

    GTCRN is an ultra-lightweight speech enhancement model designed for low-resource environments, providing high-quality noise reduction with minimal parameters and computational overhead.

  • Project
    Audio8-AI/Audio8_TTS

    A compact 0.6B-parameter multilingual text-to-speech model featuring zero-shot voice cloning and efficient deployment options for CPU and GPU.

  • Project
    kyutai-labs/unmute

    A system that enables text LLMs to listen and speak by wrapping them in low-latency speech-to-text and text-to-speech models.

  • Project
    neuphonic/neutts

    A collection of open-source, on-device text-to-speech models that enable natural-sounding speech and instant voice cloning locally on mobile and embedded devices.

  • Project
    KoljaB/RealtimeTTS

    A Python library for low-latency text-to-speech that converts strings and LLM token streams into audio using a wide variety of local and cloud engines.