sgl-project/sglang-omni

High Performance Serving Runtime For Interactive Super Intelligence System

What it solves

SGLang-Omni provides a high-performance serving runtime for complex multimodal models, specifically those focusing on speech, text-to-speech (TTS), and "omni" models (multimodal in, multimodal out). It addresses the challenge of multi-stage decoding, where a single request must pass through different types of compute patterns (like encoders, autoregressive engines, and vocoders) that have varying resource needs and scheduling requirements.

How it works

The system treats model generation as a series of coordinated stages (preprocessing, encoders, autoregressive engines, talkers, decoders, vocoders, and aggregators). It uses a specialized scheduler for each stage to match the specific workload. A control plane manages the requests while a relay data plane moves tensor payloads across various backends (shared-memory, NCCL, NIXL, and Mooncake). It integrates with SGLang for high-performance autoregressive scheduling and provides an OpenAI-compatible API surface for multimodal chat, speech generation, and transcription.

Who it’s for

It is designed for developers and researchers building applications that require low-latency, high-throughput serving of speech-to-speech or text-to-speech models, as well as those deploying omni-models on NVIDIA CUDA, Apple Silicon, or Intel GPUs.

Highlights

  • Broad Model Support: Serves a wide array of models including Qwen3-Omni, MiniMax Music 3, and various TTS and ASR (Automatic Speech Recognition) models like MOSS-TTS and Nemotron 3.5 ASR.
  • Multi-stage Runtime: Optimized for heterogeneous compute patterns across the generation pipeline.
  • OpenAI-Compatible: Offers standard endpoints for speech generation, batch/streaming speech, and transcriptions.
  • Cross-Hardware Support: Supports NVIDIA CUDA, Apple Silicon (via MLX/Torch MPS), and Intel GPUs (XPU).

Related

  • Project
    vllm-project/vllm-omni

    vLLM‑Omni is an open‑source extension of the vLLM inference engine that adds fast, scalable serving for multimodal models (text, image, audio, video, and action). It supports both autoregressive and diffusion‑style architectures, provides a full‑duplex realtime API, works across GPUs, NPUs, and other accelerators, and ships ready‑to‑run recipes for many popular Hugging Face models. The project is production‑grade, Apache‑2.0 licensed, and backed by documentation, a Slack community, and recent technical reports.

  • Project
    sgl-project/sglang

    An open-source inference framework optimized for fast serving of LLMs, vision-language, and diffusion models, particularly for agentic workloads and RL rollouts.

  • Project
    datascale-ai/opentalking

    An open-source orchestration framework for real-time digital-human conversations, integrating LLMs, TTS, STT, and video rendering via WebRTC.

  • Project
    xorbitsai/inference

    A versatile model serving library that simplifies the deployment of language, speech, and multimodal models across heterogeneous hardware.

  • Project
    OpenMOSS/MOSS-TTS

    A family of open-source speech and sound generation models providing high-fidelity TTS, multi-speaker dialogue, voice design, real-time streaming, and sound effect generation.