Aratako/Irodori-TTS

Irodori‑TTS – Flow‑Matching Text‑to‑Speech

What it is – A research‑grade TTS system that generates Japanese speech from text using a flow‑matching diffusion model (Rectified Flow Diffusion Transformer). It works on continuous latent representations produced by the DACVAE codec, following the design of Echo‑TTS.

Key capabilities

• Zero‑shot voice cloning – give a short reference recording and the model mimics that speaker.
• Multi‑modal “VoiceDesign” – combine plain text, a reference voice, and an optional caption (a short description of the desired voice style) to control speaker identity, emotion, and speaking style. Captions can be plain text or emoji‑based cues.
• Automatic duration prediction – the model estimates how long the output should be, so you usually don’t need to specify a length.
• PEFT LoRA fine‑tuning – lightweight adapters can be trained on top of the released checkpoint to adapt the model to new data without full retraining.
• Speaker‑inversion – train a reusable speaker‑embedding file that can be swapped in at inference time, avoiding the need to upload reference audio each call.
• MeanFlow distilled variant – a distilled checkpoint (v4‑Small‑MF) that can synthesize with as few as four diffusion steps, dramatically speeding up generation.
• Quantized checkpoints – int8/int4/float8 weight‑only variants are provided for low‑memory inference.

How to use it

1. Install – clone the repo and install the appropriate PyTorch extra with uv sync --extra cu128 (or rocm, xpu, cpu).
2. Run inference – a single‑line command (infer.py) lets you specify a Hugging Face checkpoint, the input text, an optional reference wav (or embedding), and an output path.
3. Web UI – two Gradio apps are shipped:
   • gradio_app.py for basic cloning / speaker‑inversion.
   • gradio_app_voicedesign.py for the caption‑plus‑reference “VoiceDesign” mode.
4. Training / fine‑tuning – scripts (train.py, prepare_manifest.py) support full‑model fine‑tuning, LoRA adapters, and speaker‑inversion training. Multi‑GPU DDP is enabled via torchrun.
5. Conversion – after training you can convert checkpoints to a pure inference‑only .safetensors file.

Model architecture (v4.1‑Small)

• Shared ModernBERT encoder – processes both the main text and any caption.
• Reference latent encoder – encodes DACVAE latents from up to 120 s of reference audio.
• Condition projectors – map text and caption embeddings into the diffusion model’s conditioning space.
• Diffusion Transformer (DiT) – joint‑attention blocks with Low‑Rank AdaLN, half‑RoPE, and SwiGLU MLPs.
• Duration predictor – built‑in module that estimates output length from the conditioning inputs.

Where to get the model – Pre‑trained checkpoints and audio samples are hosted on Hugging Face (e.g., Aratako/Irodori‑TTS‑v4.1‑Small). A demo Space shows the model in action.

Typical use‑cases

• Generating natural‑sounding Japanese speech for virtual assistants, audiobooks, or game characters.
• Rapid prototyping of new speaker voices via zero‑shot cloning.
• Style‑controlled narration by providing a textual caption (e.g., “calm female voice” or an emoji sequence).
• Deploying a lightweight, quantized TTS service on edge GPUs.

Related projects – The repo mentions an OpenAI‑compatible server (Irodori‑TTS‑Server) for easy API exposure, and it builds on Echo‑TTS and Facebook’s DACVAE codec.

---

All details are taken directly from the repository’s README; no additional features are inferred.