vllm-project/vllm-omni
A framework for efficient model inference with omni-modality models
What it solves
vLLM-Omni extends the original vLLM framework to support the serving of omni-modality models. While vLLM was primarily designed for text-based autoregressive generation, vLLM-Omni enables the efficient inference and serving of models that process and generate text, images, audio, video, and robot actions.
How it works
The framework uses a fully disaggregated architecture based on an "OmniConnector" and dynamic resource allocation across different stages of the model pipeline. It leverages vLLM's efficient KV cache management for autoregressive tasks and introduces pipelined stage execution overlapping to increase throughput. It also extends support beyond autoregressive models to include Diffusion Transformers (DiT) and other parallel generation models, allowing for heterogeneous outputs (e.g., combining text and image generation).
Who it’s for
Developers and researchers who need to deploy high-performance, production-ready serving stacks for multimodal models, including TTS, image/video generation, and robot-policy models.
Highlights
- Broad Modality Support: Handles text, image, audio, video, and action data.
- Hardware Agnostic: Supports CUDA, ROCm, MUSA, NPU, and XPU backends.
- Model Compatibility: Integrates with popular Hugging Face models like Qwen3-Omni, Cosmos3, and FLUX.
- Distributed Inference: Supports tensor, pipeline, data, and expert parallelism.
- Production Ready: Provides an OpenAI-compatible API server and streaming outputs.