vLLM-Omni Optimizations for Qwen3-Omni-30B-A3B-Instruct Serving

TL;DR

vLLM-Omni serves Qwen3-Omni-30B-A3B-Instruct as a staged pipeline of Thinker, Talker, and Code2Wav, and improves online serving performance by applying stage‑level batching, CUDA Graph capture, async chunk handoffs, async output, stage replicas for Talker/Code2Wav, and hot‑path cleanup, resulting in higher request throughput, lower audio time‑to‑first‑packet, and lower real‑time factor.

Qwen3-Omni Pipeline in vLLM-Omni

Qwen3-Omni combines multimodal understanding with speech generation and is served in vLLM-Omni as a three‑stage dataflow: Thinker performs multimodal reasoning and text generation, Talker converts hidden states to RVQ codec codes, and Code2Wav reconstructs waveform audio from those codes. The pipeline uses the OpenAI‑compatible /v1/chat/completions endpoint, with the modalities field in the request body specifying output types such as ["text" ] or ["text", "audio" ]. The default deploy profile is selected automatically when launching with --omni; an explicit profile can be supplied via --deploy-config vllm_omni/deploy/qwen3_omni_moe.yaml. The same launch command works across CUDA, NPU, ROCm, and XPU backends because vLLM-Omni merges runtime‑specific deltas from the platform section of the profile.

Optimization Techniques

Stage Decomposition and Batching

Stage decomposition separates Thinker, Talker, and Code2Wav into independent serving objects, allowing each stage to have its own batching, graph, and device policy instead of being forced to share a single policy that lets the slowest sub‑path gate the rest. Per‑stage batching groups concurrent requests into one Talker MTP invocation and one Code2Wav forward, filling SMs that would otherwise be idle due to single‑request micro‑work and amortizing fixed per‑step costs across the batch. This batched, stage‑decomposed configuration forms the Batch baseline for all subsequent optimizations.

CUDA Graph Capture

CUDA Graph removes repeated CPU‑side kernel dispatch on every decode step by capturing a fixed operator sequence once and replaying it with minimal CPU work. Thinker and Talker use vLLM's outer CUDA Graph path; Talker's inner code predictor is optimized with torch.compile (no second graph layer to avoid conflict); Code2Wav uses an inner CUDAGraphDecoderWrapper that captures shapes based on codec_chunk_frames and codec_left_context_frames from the connector config, precomputes SnakeBeta caches, and dispatches chunks via the wrapper’s batched or chunked decode entry points. Enabling CUDA Graph on all three stages raises request throughput from 2.2 to 8.6 req/s (+299%), cuts mean audio TTFP from 5884 ms to 2790 ms (−53%), and reduces mean audio RTF from 1.15 to 0.59 (−49%) at concurrency 64.

Async Chunk Handoffs

Async chunk replaces full‑payload stage barriers with pipelined partial handoffs. Thinker emits embedding rows incrementally; Talker accumulates codec frames and slices them on initial_codec_chunk_frames / codec_chunk_frames boundaries; the async scheduler overlaps chunk transfer with stage compute, allowing each stage to start work while the previous one is still decoding. This change yields the largest single reduction in audio TTFP, dropping mean audio TTFP from 2790 ms (CUDA Graph) to 655 ms (−77%) while request throughput increases modestly from 8.6 to 9.3 req/s (+8%) at concurrency 64; mean audio RTF remains at 0.63.

Async Output

Async output decouples payload construction from stage handoff by moving the assembly of Thinker connector payloads to a non‑blocking path, so decode workers are not stalled by synchronous copying of embeddings and hidden states. With async chunk already enabled, async output keeps mean audio TTFP near 631 ms (−4% from async chunk) while lowering mean audio RTF from 0.63 to 0.47 (−25%) and increasing request throughput from 9.3 to 11.3 req/s (+22%) at concurrency 64.

Stage Replicas

Stage replicas add capacity only to the stages that saturate under load. Because each request requires hundreds of Talker decode steps and Code2Wav vocoder forwards but only a single Thinker generation, Talker and Code2Wav become the bottleneck first. Deploying 2× Talker and 2× Code2Wav replicas on GPUs 1 and 2 while keeping a single Thinker on GPU 0 absorbs the speech‑side backlog without duplicating the large multimodal Thinker. Adding replicas on top of async output raises request throughput to 11.7 req/s (+4% from async output) at concurrency 64, with mean audio TTFP staying near 632 ms and mean audio RTF at 0.47.

Hot‑Path Cleanup

Hot‑path cleanup removes per‑step overhead that scales with utterance length in the Talker decode loop and connector payloads. Changes include: sending only the new embed.decode row after the initial prefill (O(1) per‑step connector traffic), using the single‑GPU uni executor by default to avoid multiprocess overhead, eliminating repeated torch.cat in payload construction, rewriting the Talker code predictor to use re‑prefill with SDPA, native GQA, inline top‑k sampling, cached module references, and torch.compile (no conflicting second graph layer), keeping intermediate tensors GPU‑resident in model_intermediate_buffer, skipping redundant device‑to‑host reads, and avoiding unnecessary multimodal position computations. In a long‑context single‑request test, these changes reduced end‑to‑end latency from 21.28 s to 7.37 s, audio TTFP from 3197 ms to 1796 ms, and audio RTF from 0.71 to 0.28. These gains stack with the optimizations above and are reflected in the DFX perf suite baselines.

Validation Results

Validation used a controlled benchmark sweep on Seed‑TTS en with the model Qwen3-Omni-30B-A3B-Instruct, prompt lengths of 10/160/320/640 tokens, concurrency levels of 1/16/32/64, five warmups, and three visible GPUs mapped as 0/1/2. Each configuration restarted the server with an isolated deploy profile and added one optimization on top of the previous row. Batch through Async output pinned one stage per GPU (Thinker/Talker/Code2Wav on GPUs 0/1/2, single replica each); the Stage replicas row kept Thinker on GPU 0 and ran 2× Talker + 2× Code2Wav on GPUs 1 and 2.

At concurrency 64:

• Batch baseline: 2.2 req/s, mean audio TTFP 5884 ms, mean audio RTF 1.15
• • CUDA Graph: 8.6 req/s (+299%), TTFP 2790 ms (−53%), RTF 0.59 (−49%)
• • Async chunk: 9.3 req/s (+8%), TTFP 655 ms (−77%), RTF 0.63
• • Async output: 11.3 req/s (+22%), TTFP 631 ms (−4%), RTF 0.47 (−25%)
• • Stage replicas: 11.7 req/s (+4%), TTFP 632 ms, RTF 0.47

Throughput (Figure 7) shows request throughput rising from the Batch baseline of 2.2 req/s to 11.7 req/s at concurrency 64 (5.4×) and from 1.1 to 6.8 req/s at concurrency 32. The largest single jump is CUDA Graph (4×); async output provides the final push at high concurrency, and stage replicas deliver the peak throughput with headroom that grows as concurrency increases.

Real‑time factor (Figure 8) drops from an above‑real‑time 1.15 under Batch to 0.47 at concurrency 64, indicating decode moves from lagging playback under load to running comfortably ahead of it.

First‑packet latency (Figure 9) falls from ~5884 ms (Batch) to ~632 ms at concurrency 64, with async chunk contributing the largest single cut to ~655 ms and later layers preserving that gain.

Deployment Quickstart

To serve Qwen3-Omni-30B-A3B-Instruct with the default omni profile:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --port 8091

For explicit configuration, supply the staged deploy profile:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --port 8091 \
  --deploy-config vllm_omni/deploy/qwen3_omni_moe.yaml

The launch command works unchanged across CUDA, NPU, ROCm, and XPU because vLLM-Omni automatically merges the matching platform deltas from the profile’s platforms: section.

Requests should be sent to /v1/chat/completions. Set the modalities field in the request body to declare output types: ["text" ] for text only, or ["text", "audio" ] for text plus speech.

For details on async‑chunk settings, multi‑replica layouts, and further deployment options, consult the Qwen3‑Omni online serving guide at https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/qwen3_omni/README.md.

Acknowledgements

The post thanks the Qwen3‑Omni contributors in vLLM‑Omni, listing Haiyan Wu, Taichang Zhou, Canlin Guo, Ruirui Yang, Ziming Huang, Wengang Zheng, Lianhao Xu, Han Gao, Junhong Liu, Samit Huang, Hao Chen, Alex Brooks, Chenguang Zheng, Peiqi Yin, Wenjing Chen, Nick Cao, Shunyang Li, Yong Yang, Divyansh Singhvi, Yueqian Lin, Dayu Qiu, Roger Wang, and Hongsheng Liu for their contributions and feedback.

References

• Qwen3‑Omni pipeline topology: pipeline.py
• Qwen3‑Omni model wrapper: qwen3_omni.py
• Qwen3‑Omni stage input processors: stage_input_processors/qwen3_omni.py
• Qwen3‑Omni deploy profile: qwen3_omni_moe.yaml
• Qwen3‑Omni async‑chunk perf config: test_qwen3_omni_async_chunk.json
• Qwen3‑Omni multi‑replica perf config: test_qwen3_omni_multi_replicas.json
• Qwen3‑Omni model repository: Qwen/Qwen3-Omni-30B-A3B-Instruct
• Optimization pull requests: CUDA Graph (Thinker #523, Talker #669, Code2Wav #2376); Async chunk (cross‑stage #727, async scheduling #951, inter‑packet latency #1656); Async output (#4476); Stage replicas (multi‑stage #2396, runtime and control plane #3855); Hot‑path cleanup (#3007, #3164, #3878)
• For interest in Qwen3‑Omni serving or omni‑modality inference, join the #sig-omni channel in vLLM Slack or open an issue in the vLLM‑Omni GitHub repository.