Phonon TTS Release: High-Quality On-Device Speech Synthesis

Overview

Phonon is a compact, on-device text-to-speech (TTS) model designed to operate without server round-trips or network connectivity. By utilizing flow matching techniques—derived from the Pocket TTS lineage—Phonon achieves high-quality audio synthesis within a small memory footprint, making it suitable for deployment on mobile devices, laptops, and web browsers.

Technical Architecture and Capabilities

Phonon is available in two primary sizes: 65 MB and 75 MB. The models support five languages: English, French, German, Spanish, and Portuguese.

  • Inference Efficiency: The model is optimized for speed, topping the Hugging Face Open TTS Leaderboard for CPU streaming, specifically in time-to-first-audio (TTFA) metrics. It achieves a TTFA of 28 ms for the 31m version and 38 ms for the 40m version.
  • Inference Frameworks: The team developed ptts, a dedicated inference framework supporting over 10 backends, including CPU (Intel, Arm, WebAssembly), GPU (CUDA, Vulkan, Metal, WebGPU), and NPU (Apple Core ML, Snapdragon).
  • Transformers Integration: Phonon is integrated into Hugging Face Transformers (v5.21 and later). It utilizes a modular design where the model processes audio latents autoregressively. The audio codec acts as a tokenizer, and voices are represented as tensors of speaker latents rather than raw audio recordings.

Licensing and Accessibility

  • Phonon 31m: Licensed under Apache 2.0, allowing for free use in commercial products.
  • Phonon 40m: Licensed under CC BY-NC, intended for non-commercial projects.
  • Deployment: The models support voice cloning and custom voice design via Gradium Studio, enabling developers to integrate personalized voices into local applications.

Performance and Benchmarking

Phonon has been evaluated on the Hugging Face Open TTS Leaderboard, which assesses models on intelligibility (Word Error Rate), speed (inverse real-time factor and TTFA), and naturalness (UTMOS/KOMOS).

  • Intelligibility: Phonon ranks first in German and Spanish and maintains a top-three position across all evaluated languages, performing competitively against models significantly larger in size.
  • Speed: It currently leads the CPU streaming benchmark for time-to-first-audio, demonstrating its suitability for real-time, low-latency applications.

Implementation in Transformers

Developers can use Phonon within the Transformers ecosystem by leveraging the GradiumPhononTTSForConditionalGeneration class. The model uses a chat template format where audio prompts are injected as tensors. 

from transformers import AutoProcessor, GradiumPhononTTSForConditionalGeneration

model_id = "gradium/phonon-31m-en"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = GradiumPhononTTSForConditionalGeneration.from_pretrained(model_id, device_map="auto")

# The processor handles audio tokenization and chat template formatting
inputs = processor.apply_chat_template(
    conversation,
    tokenize=True,
    return_dict=True,
    add_generation_prompt=True,
    processor_kwargs={"audio_prompt_latents": [voice]},
).to(model.device)

outputs = model.generate(**inputs)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)

This implementation allows for batched generation and full-graph torch.compile support, facilitating the synthesis of large speech datasets at scale.

Sources