vllm-project/vllm-omni

A framework for efficient model inference with omni-modality models

What it solves

vLLM-Omni extends the original vLLM framework to support "any-to-any" multimodal model serving. While vLLM was primarily designed for text-based autoregressive generation, vLLM-Omni enables the serving of models that process and generate text, images, audio, video, and robot actions, supporting both autoregressive and non-autoregressive (such as Diffusion Transformers) architectures.

How it works

The framework uses a fully disaggregated serving architecture based on an OmniConnector and dynamic resource allocation across different stages. It leverages vLLM's efficient KV cache management for autoregressive tasks and implements pipelined stage execution to overlap processing and increase throughput. It also supports distributed inference through tensor, pipeline, data, and expert parallelism.

Who it’s for

Developers and researchers who need to deploy and serve high-performance multimodal models, including TTS, diffusion models for image/video/audio generation, and robot-policy models, using an OpenAI-compatible API server.

Highlights

  • Broad Modality Support: Handles text, image, audio, video, and action data.
  • Heterogeneous Architectures: Supports both autoregressive models and Diffusion Transformers (DiT).
  • Real-time Capabilities: Offers full-duplex real-time serving with streaming audio input and output.
  • Wide Hardware Compatibility: Supports CUDA, ROCm, XPU, NPU, and MUSA.
  • Model Integration: Seamlessly integrates with popular Hugging Face models like Qwen3-Omni, MiniMax H3, and various robot-policy models.

Related

  • Project
    sgl-project/sglang-omni

    A multi-stage serving runtime for omni, speech, and TTS models that optimizes generation across heterogeneous compute stages.

  • Project
    verl-project/verl-omni

    A general RL training framework for multimodal generative models, providing fast rollouts and stable post-training for diffusion and omni-modality models.

  • Project
    EvolvingLMMs-Lab/lmms-engine

    A unified, high-performance training engine for multimodal models that integrates distributed training, kernel fusion, and advanced optimizers to enable scalable pretraining and fine-tuning.

  • Project
    sgl-project/sglang

    A high-performance serving framework for LLMs and multimodal models that provides low-latency, high-throughput inference across single GPUs and large distributed clusters.

  • Project
    ParisNeo/lollms-webui

    A unified local web interface for accessing hundreds of LLMs and multimodal AI models for text, image, video, and music generation.