vllm-project/vllm-omni

vLLM‑Omni – Unified Serving for Omni‑Modality Models

What it is – vLLM‑Omni is an open‑source extension of the popular vLLM inference engine. It adds a unified runtime that can serve any‑to‑any multimodal model: text, images, audio, video, and even action/robot‑policy outputs. The project ships a production‑ready server, a set of “recipes” for popular models, and a full‑duplex realtime API that can accept streaming inputs (e.g., microphone audio) while streaming outputs back.

Why it matters – Traditional LLM serving frameworks handle only autoregressive text generation. Modern foundation models are increasingly multimodal (e.g., Qwen‑3‑Omni, MiniCPM‑o, diffusion transformers). vLLM‑Omni bridges that gap by:

• Re‑using vLLM’s high‑throughput KV‑cache and scheduler logic.
• Adding OmniConnector stages that can route data between heterogeneous model components (e.g., a text encoder → diffusion decoder → action policy).
• Supporting both autoregressive and non‑autoregressive architectures (Diffusion Transformers, parallel generation models).
• Providing hardware‑agnostic execution: CUDA, ROCm, XPU, NPU, and multi‑node distributed parallelism.

Key capabilities (as described in the README)

Capability	Details
Omni‑modality support	Text, image, audio, video, and action generation in a single server.
Model catalog	Works out‑of‑the‑box with many Hugging‑Face models: Qwen3‑Omni, MiniCPM‑o 4.5, Cosmos3, BAGEL, TTS models (Qwen3‑TTS, Breeze‑TTS‑2), diffusion models (MiniMax H3, LingBot World), robot‑policy models (π0, DreamZero‑DROID).
Full‑duplex realtime API	Streaming audio in and out, enabling interactive voice assistants or robot control.
Dynamic resource allocation	Stages are disaggregated; the runtime can move work across GPUs, NPUs, or CPUs on‑the‑fly.
Parallelism	Tensor, pipeline, data, and expert parallelism for large models.
OpenAI‑compatible server	Drop‑in replacement for OpenAI’s chat/completion endpoints.
Extensible recipes	Ready‑made deployment scripts ("recipes") for dozens of models, covering quantization, caching, and hardware‑specific tweaks.
Cross‑stage KV & multimodal payload transfer	Efficient hand‑off between stages (e.g., AR‑to‑DiT handoff for image generation).

Typical use cases

• Building a multimodal AI assistant that can see images, listen to speech, and respond with text, audio, or generated images/video.
• Deploying real‑time diffusion generation (image/video/audio) at production scale, with request‑level batching.
• Running robotic policy inference where the model receives sensor streams and outputs action commands.
• Providing a single API endpoint for a suite of foundation models, simplifying client‑side integration.

Getting started

1. Install – Follow the docs (pip install or Docker). Nightly CUDA images are available on Docker Hub (vllm/vllm-omni).
2. Run the server – vllm-omni serve --model Qwen3-Omni --api-key <key> starts an OpenAI‑compatible endpoint.
3. Try a recipe – The recipes/ folder contains ready‑to‑run configs, e.g., recipes/OpenBMB/MiniCPM-o-4_5.md for the MiniCPM‑o model.
4. Explore the docs – Full guides for installation, quickstart, and the realtime duplex API are on https://vllm-omni.readthedocs.io.

Community & support

• Documentation – https://vllm-omni.readthedocs.io
• User forum – https://discuss.vllm.ai
• Slack – https://slack.vllm.ai (channel #sig-omni)
• WeChat – QR code in the repo assets.
• DeepWiki – curated deep‑dive articles.
• Citation – If you use the library in research, cite the two arXiv reports listed in the README.

License – Apache License 2.0.

---

In short, vLLM‑Omni turns the fast, scalable text‑only vLLM engine into a general‑purpose multimodal serving platform, letting developers and researchers deploy cutting‑edge foundation models for text, vision, audio, video, and robot actions with a single, high‑performance runtime.