vLLM-Omni TTS Inference Engineering

TL;DR

vLLM-Omni optimized TTS inference for multiple models by addressing stage-specific bottlenecks in the Talker and Code2Wav stages, achieving up to 172% higher audio throughput and nearly halving end-to-end latency for Qwen3-TTS at high concurrency.

How TTS Inference Differs from Traditional LLM Inference

TTS inference uses autoregressive models but faces different serving bottlenecks than text-only LLMs due to its multi-stage pipeline (Talker and Code2Wav) and strict latency budgets for streaming audio output, where chunk size affects both first-packet latency and cross-chunk audio quality.

Optimization Overview

vLLM-Omni selects optimizations based on each TTS model's pipeline structure, decode state, batch shapes, and numerical constraints rather than applying a fixed recipe, as techniques like stage separation, batched preprocessing, torch.compile, and GPU-resident state only benefit specific architectures.

Qwen3-TTS: A Full Optimization Path

For Qwen3-TTS, decoupling connector chunks from the Code2Wav decode window, batching Stage 0 preprocessing, cleaning up hot-path overheads, and aligning numerical precision to fp32 improved audio throughput by 61.5% and reduced P99 end-to-end latency by nearly half on H20 × 2 at c=64.

1. Streaming: Decoupling Connector Chunks from the Code2Wav Decode Window

Decoupling connector streaming chunk size (codec_chunk_frames) from Code2Wav's internal decode window (decode_chunk_frames and decode_left_context_frames) allows independent tuning: small connector chunks reduce first-packet latency while Code2Wav maintains a 300-frame decode window plus 25 frames of left context for cross-chunk audio continuity.

2. Throughput: Stage 0 Decode Preprocessing

Batching Talker decode preprocessing (speaker embedding preparation, trailing_text maintenance, input embedding construction) eliminated per-request Python overhead in the decode hot path, reducing GPU idle time caused by small tensor allocations and kernel launches at high concurrency.

3. Hot-Path Cleanup

Replacing O(N²) req_id_to_index lookups with a dictionary, skipping non-streaming paths early, precomputing codec-disallowed masks, and tuning CUDA Graph capture for Code2Wav reduced Python overhead in the frequent c=64 decode loop without changing model computations.

4. Numerical Precision: fp32 Alignment for the Code Predictor

Splitting the Talker code predictor to keep RMSNorm variance, RoPE cos/sin, attention, and QKV projection in fp32 via PyTorch-native implementations prevented precision drift from bfloat16 fused kernels in short-sequence, high-frequency autoregressive steps.

5. Validation

After stacking optimizations, Qwen3-TTS audio throughput on H20 × 2 increased from 26.55 to 42.88 audio-s/s (+61.5%) and P99 end-to-end latency dropped from 17.7s to 9.0s at c=64 for voice cloning, with warm concurrency sweets showing non-linear E2E growth due to amortized fixed costs.

VoxCPM2: Single-Stage Hybrid TTS

For VoxCPM2, whole-forward torch.compile reduced Python-to-compiled boundaries in the MiniCPM4 Talker, and batching the CFM/LocDiT decode tail across requests increased audio throughput by 172.0% on H20 × 1 at c=64.

Exploring torch.compile

Wrapping the entire Model.forward in torch.compile with fullgraph=False let Dynamo optimize the 28-layer MiniCPM4 loop despite PagedAttention breaks, reducing cudaLaunchKernel count by ~71% and kernel time by ~27%, while per-layer compile alone failed to drop launch count due to unresolved boundaries.

CFM/LocDiT Decode-Tail Batching

Batching lm_h, residual outputs, and prefix feature conditions across requests for CFM/LocDiT, feat_encoder, and stop_head before scattering results back, combined with sliding-window VAE decoding and fused operations, turned tiny per-request diffusion workloads into efficient GPU batches, boosting H20 × 1 throughput from 4.19 to 10.83 req/s (+158.8%) and audio throughput from 12.16 to 33.07 audio-s/s (+172.0%) at c=64.

Higgs Audio V3: Dynamic Batches and Multi-Codebook State

For Higgs Audio V3, moving multi-codebook decode state to GPU-resident batched tensors and using local MLP CUDA Graph (instead of PIECEWISE) avoided Python overhead and synchronization, achieving 35.26 audio-s/s throughput on a single H20 at c=16.

Moving Decode State to the GPU

Converting per-request Python dict state (_decode_last_codes, _decode_has_codes, delay count, EOC countdown, etc.) to GPU-resident batched tensors eliminated Python loops and D2H synchronization in the decode hot path, with state updates now occurring on the batched GPU trajectory.

Adapting CUDA Graph to Dynamic Batch Shapes

Using a uniform single-token decode batch for CUDA Graph capture (where decode_mask is all True) avoided shape mismatches from the audio feedback mechanism's boolean mask, ensuring stable graph shapes despite dynamic batching in the scheduler.

Local MLP CUDA Graph vs. PIECEWISE

Local MLP CUDA Graph covering post_attention_layernorm + mlp outperformed PIECEWISE graph for Higgs v3 because the model's multi-codebook delay pattern causes data-dependent embedding lookups and pre-attention index operations that break larger graphs or require costly synchronization.

A Rejected Staging-Overlap Design

A one-step audio staging overlap design to hide D2H copies was rejected as structurally unsafe under dynamic batching, since scheduler-induced request reordering or completion could break cursor-to-request mappings, making the approach unreliable without request-id keying and drain hooks.

Fish Speech S2 Pro: When Generic Attention Becomes the Bottleneck

For Fish Speech S2 Pro, a model-specific q_len=1 attention kernel and Fast AR buffer reuse addressed GPU-side bottlenecks from generic attention overhead and repeated allocations, achieving 23.72 audio-s/s throughput on H20 at c=64.

Model-Specific Attention Kernel

A Fish-specific Triton kernel for SlowAR decode attention (handling q_len=1, fp16/bf16, head_dim=128, block size 16, GQA layout) replaced generic paged/varlen attention for pure decode steps, with split-partial-combine for long sequences and CPU-side upper bounds to avoid synchronization during path selection.

Fast AR Buffer Reuse and Compile

Preallocating and reusing _embed_buf, _k_cache, and _v_cache tensors for Fast AR eliminated repeated allocation in short-sequence decode steps, while torch.compile with fullgraph=False and dynamic=True memoized subgraphs for the four-layer transformer despite SDPA internal breaks.

DAC and Runtime-Side Optimizations

Switching codec payload transfer from Python list[int] to tensor serialization, enabling fp16 DAC support, implementing frame-count-batched DAC batching, and overlapping connector transfer with DAC computation via async chunk processing reduced allocation, GC pressure, and blocking at high concurrency.

Performance Data

The optimizations delivered measurable gains across models, as validated in vLLM-Omni cookbook benchmarks.

Qwen3-TTS (c=64, p=512, H20 × 2, voice clone)

Metric	Before	After	Change
Audio throughput	26.55 audio-s/s	42.88 audio-s/s	+61.5%
Median E2EL	9654ms	5699ms	−41.0%
P99 E2EL	17686ms	8956ms	−49.4%
P99 TTFP	7558ms	5563ms	−26.4%

VoxCPM2 (c=64, H20 × 1, before/after CFM batching)

Metric	Before	After	Change
Request throughput	4.19 req/s	10.83 req/s	+158.8%
Audio throughput	12.16 audio-s/s	33.07 audio-s/s	+172.0%

Fish Speech S2 Pro (H20, single GPU, c=64, Triton KV cache + tensor payload)

Metric	Value
Audio throughput	23.72 audio-s/s
Request throughput	5.95 req/s
Mean TTFP	899.67 ms
Mean E2EL	10.47 s

Higgs Audio V3 (H20, single GPU, c=16, eager + local MLP graph)

Metric	Value
Request throughput	5.18 req/s
Audio throughput	35.26 audio-s/s
Wall time	96.5s
Speedup vs. baseline	2.70×