andimarafioti/faster-qwen3-tts

What it solves

Faster Qwen3-TTS addresses the high latency and slow inference speeds of the standard Qwen3-TTS model. It enables real-time text-to-speech generation, significantly reducing the Time to First Audio (TTFA) and increasing the Real-Time Factor (RTF) for both streaming and non-streaming audio generation.

How it works

The project optimizes inference using torch.cuda.CUDAGraph to capture execution graphs, bypassing the need for complex dependencies like Flash Attention, vLLM, or Triton. It supports two primary backends:

• Torch/CUDA-graph: The default high-performance path for NVIDIA GPUs.
• GGML: An experimental adapter for the qwentts.cpp runtime, allowing for quantized inference.

To maintain audio quality during streaming, it uses a sliding window with 25-frame left context to decode audio chunks and avoid boundary artifacts. It also automatically appends silence to reference audio in In-Context Learning (ICL) mode to prevent phoneme bleeding at the start of generated speech.

Who it’s for

• Developers building real-time AI voice applications.
• Users wanting to run Qwen3-TTS on NVIDIA GPUs (including Jetson AGX Orin) with minimal latency.
• Those seeking an OpenAI-compatible TTS API for integration with tools like OpenWebUI.

Highlights

• Significant Speedups: Achieves up to 9.8x RTF improvement and 6.5x TTFA reduction on various NVIDIA GPUs.
• Flexible Generation Modes: Supports voice cloning (via x-vector or ICL), predefined speaker IDs (CustomVoice), and instruction-based voice design.
• Streaming Support: Yields audio chunks during generation for immediate playback.
• OpenAI Compatibility: Includes an API server that follows the POST /v1/audio/speech contract.
• Multiple Backends: Offers both a native PyTorch path and an experimental GGML backend.