Qwen3-TTS Speed and Cost Optimization for Sub-50ms Latency

Nari Labs has developed a custom implementation of the Qwen3-TTS 1.7B CustomVoice model that achieves a p95 time-to-first-audio (TTFA) of under 50 milliseconds while maintaining real-time playback at 10 requests per second (RPS) on a single NVIDIA H100 SXM. This optimization significantly reduces the cost of high-performance text-to-speech, estimating a cost of approximately $2 per 1 million characters at full utilization.

Benchmarking Real-Time TTS Performance

To evaluate real-time viability, Nari Labs defines a successful TTS server based on four criteria: low audible TTFA, zero audio underruns (where the client runs out of buffered audio), high capacity as RPS increases, and intelligible output.

In benchmarks comparing five implementations—including vLLM-Omni, SGLang-Omni, VoxServe, and M*—the Nari Labs implementation was the only one to maintain sub-50ms p95 TTFA through 10 RPS. While other engines like VoxServe could reach sub-50ms at 1 RPS, their latency spiked significantly as load increased; for example, at 6 RPS, most engines exceeded 100ms p95 TTFA.

Addressing Common Latency Bottlenecks

Two primary optimizations were applied across all tested engines to establish a fair baseline:

Leading Silence Removal: Models often produce tens of milliseconds of silence before the first sound. Nari Labs implemented a dynamic trim that detects sustained speech via RMS windows and removes leading silence, improving TTFA by approximately 80ms.
Frame Accumulation Tuning: The system balances the number of codec frames collected before decoding. Smaller initial chunks reduce TTFA but increase decoder overhead; larger chunks improve batching and playback stability. The optimal configuration uses small initial chunks that ramp up in size for subsequent output.

Architectural Optimizations for Qwen3-TTS

Qwen3-TTS uses a hierarchical multi-codebook generation process involving three modules: the Talker (predicts the first codebook token), the Code Predictor (generates the remaining 15 tokens), and the Codec (converts tokens to waveforms).

Unified Scheduling Surface

Rather than splitting the process into two stages (generation and decoding), Nari Labs treats the Talker, Code Predictor, and Codec as three independently schedulable tasks under a single scheduler. This allows the system to rearrange work based on urgency—such as prioritizing a Codec job approaching a playback deadline over a Talker job—and enables more granular batching of requests waiting for the same module.

Urgency-Based Scheduling

The scheduler distinguishes between two types of urgency:

Initial Latency: Requests that have not yet produced their first audio chunk are given the highest priority to minimize TTFA.
Playback Deadlines: Once playback begins, chunks are prioritized only as they approach the deadline required to prevent underruns.

To maintain GPU efficiency, the scheduler selects an urgent request as an "anchor" and fills the remainder of the batch with compatible, non-urgent work.

Code Predictor and Codec Enhancements

CUDA Graphs and Triton Kernels: Because the Code Predictor performs a fixed number of steps (15) per frame, Nari Labs captured the entire loop as a single CUDA graph and utilized a specialized Triton attention kernel for short contexts, reducing host-driven overhead.
State-Cached Incremental Decoding: To avoid reprocessing the full frame history for every new chunk, the Codec uses a state cache to retain Transformer context and convolutional state. The system uses full decoding for the first audio chunk to minimize initialization overhead, then switches to incremental decoding for sustained playback.

Industry Perspectives and Trade-offs

Community discussion highlights several critical considerations for deploying ultra-low latency TTS in production:

The "Creepy" Valley of Speed: Some developers argue that responses under 200ms can feel unnatural or "creepy," as humans typically have a 200ms auditory processing latency. Artificial latency or guardrails may be necessary to make conversations feel human.
Quality vs. Latency: There is a noted "quality hard wall" where further latency reductions may compromise the cadence, expression, and overall naturalness of the voice.
On-Device Requirements: While H100 performance is impressive, the ultimate goal for many developers is moving these capabilities to mobile devices to eliminate cloud latency and improve privacy.
End-to-End Latency: For agentic scenarios, TTS latency is only one part of the chain. Total perceived latency includes STT (Speech-to-Text) and LLM inference. Some suggest that only models that bake speech tokens directly into the LLM (like OpenAI's GPT-4o/Realtime) can truly solve the end-to-end latency problem.

Nari Labs has open-sourced both the implementation and the benchmark suite.