andimarafioti/faster-qwen3-tts

Real-time text-to-speech with Qwen3-TTS

What it solves

Faster Qwen3-TTS addresses the high latency and slow inference speeds of the original Qwen3-TTS models. It enables real-time, high-quality text-to-speech generation with significantly lower Time to First Audio (TTFA) and higher Real-Time Factor (RTF), making it suitable for interactive applications.

How it works

The project provides two primary backends for optimized inference:

  • Torch Backend: Utilizes torch.cuda.CUDAGraph on NVIDIA CUDA and AMD ROCm to capture and replay execution graphs, drastically reducing overhead.
  • GGML Backend: An experimental adapter for the qwentts.cpp runtime, supporting CUDA, Apple Silicon Metal, and AMD HIP, allowing for efficient execution on a wider range of hardware including Macs.

It supports streaming output, where audio chunks are yielded during generation using a sliding window with 25-frame left context to prevent boundary artifacts. It also implements a fix for ICL (In-Context Learning) phoneme artifacts by appending silence to reference audio by default.

Who it’s for

  • Developers building real-time AI voice agents or interactive TTS systems.
  • Hardware enthusiasts wanting to run Qwen3-TTS on NVIDIA GPUs, AMD GPUs, or Apple Silicon Macs.
  • Researchers comparing inference optimization techniques for TTS models.

Highlights

  • Massive Speedups: Achieves up to 9.8x RTF improvement and 6.5x TTFA reduction on various GPUs (e.g., RTX 4090, H100).
  • Flexible Voice Options: Supports voice cloning (ICL and x-vector), predefined CustomVoice IDs, and instruction-based VoiceDesign.
  • Multi-Platform Support: Native support for CUDA, ROCm, Metal (Apple Silicon), and GGML.
  • Developer Tools: Includes an OpenAI-compatible API server, a minimal web UI for live metrics, and a CLI for quick generation.

Related

  • Project
    antirez/qwen-asr

    A pure C implementation of the Qwen3-ASR inference pipeline for efficient speech-to-text transcription on CPU, NVIDIA, and AMD GPUs.

  • Dispatch
    Qwen3-TTS Speed and Cost Optimization for Sub-50ms Latency

    Nari Labs has open-sourced a custom implementation of Qwen3-TTS 1.7B that achieves sub-50ms p95 time-to-first-audio (TTFA) at 10 requests per second on a single NVIDIA H100 SXM.

  • Project
    KoljaB/RealtimeTTS

    A Python library for low-latency text-to-speech that converts strings and LLM token streams into audio using a wide variety of local and cloud engines.

  • Project
    SWivid/F5-TTS

    F5-TTS is a high-performance text-to-speech system using a Diffusion Transformer and flow matching to generate fluent, faithful speech with fast training and inference.

  • Dispatch
    vLLM-Omni TTS Inference Engineering

    vLLM-Omni engineered TTS inference for models like Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro by applying model-specific optimizations that decouple latency and throughput bottlenecks, significantly improving audio throughput and reducing end-to-end latency.