inworld-ai/tts

What it solves

This project provides the training and modeling infrastructure needed to create high-quality text-to-speech (TTS) systems. It allows developers to pre-train, fine-tune, or align SpeechLM-based models using their own audio data, supporting everything from single-GPU setups to large multi-GPU clusters.

How it works

The system uses a SpeechLM architecture combined with 1D audio-codecs to convert text into speech. The workflow involves:

1. Data Preparation: Raw audio and transcripts are processed into a specific JSONL format.
2. Vectorization: Audio is converted into audio-codes using an encoder (compatible with xcodec2).
3. SFT Training: Supervised Fine-Tuning (SFT) teaches the model to generate audio codes conditioned on text.
4. RLHF Alignment: Reinforcement Learning from Human Feedback (RLHF) further refines the model using reward functions (such as Word Error Rate) to align output with human preferences.
5. Inference: A trained model, along with an encoder and decoder, generates speech from a text input and an audio prompt.

Who it’s for

AI researchers and engineers building custom text-to-speech models who need a scalable training pipeline that supports distributed training (DDP, DeepSpeed, FSDP) and RLHF alignment.

Highlights

• Distributed Training: Native support for DDP, DeepSpeed, and FSDP for scaling across multiple GPUs.
• RLHF Integration: Built-in capabilities for alignment using reward functions and vLLM for accelerated generation during training.
• Flexible Modeling: Supports arbitrary SpeechLM models and is compatible with xcodec2 checkpoints.
• Comprehensive Pipeline: Includes ready-to-use scripts for data vectorization, shard merging, and model conversion for serving.