Aratako/Irodori-TTS

A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control

What it solves

Irodori-TTS is a high-fidelity text-to-speech (TTS) system designed to generate natural-sounding audio from text. It addresses the challenge of creating flexible voice identities, allowing for zero-shot voice cloning from a short reference audio clip or detailed style control via text captions (VoiceDesign), as well as the ability to automatically estimate the length of the generated audio based on the input text.

How it works

The system uses a Flow Matching-based architecture (Rectified Flow Diffusion Transformer) that operates over continuous latents provided by a DACVAE codec. It employs several specialized encoders:

  • Text Encoder: Uses token embeddings from a pretrained LLM with transformer layers.
  • Reference Latent Encoder: Processes reference audio to condition the model on a specific speaker's identity.
  • Caption Encoder: Processes text descriptions of emotion, tone, and style to control the vocal delivery.
  • Duration Predictor: An integrated component in v3 that automatically estimates the output audio length.

Who it’s for

  • AI Audio Developers: Those looking to implement high-quality TTS with advanced conditioning (cloning and style control).
  • Content Creators: Users who need specific vocal emotions or styles for their projects without recording new audio.
  • ML Researchers: Developers interested in Flow Matching and Diffusion Transformers for audio generation.

Highlights

  • Multi-modal Voice Design: Combines text, reference speech, and captions for precise control over voice identity and emotion.
  • Zero-shot Voice Cloning: Ability to clone a voice using only a reference audio sample.
  • Emoji-based Style Control: Supports using emojis in input text to influence non-verbal vocal expressions.
  • PEFT LoRA Fine-Tuning: Supports parameter-efficient adaptation for specific voices or styles.
  • Speaker Inversion: Allows learning and saving reusable speaker embedding tokens to avoid providing reference audio for every call.
  • Automatic Watermarking: Integrates SilentCipher for automatic audio watermarking.