sgl-project/sglang-omni

SGLang-Omni is a high-performance serving framework for audio models (TTS, ASR) and unified multimodal models.

What it solves

SGLang-Omni provides a high-performance serving runtime for multimodal, speech, and text-to-speech (TTS) models. It addresses the challenge of "multi-stage decoding," where a single request requires a sequence of different compute patterns (e.g., preprocessing, autoregressive generation, and vocoding) that often have different resource needs and scheduling requirements.

How it works

The system models generation as a series of coordinated stages (such as encoders, autoregressive engines, talkers, and vocoders). It uses stage-specialized scheduling to match the workload of each stage—leveraging SGLang for autoregressive tasks while using lightweight loops for streaming vocoders. A control plane coordinates the requests, while a relay data plane moves tensor payloads across various backends like shared-memory, NCCL, and Mooncake.

Who it’s for

It is designed for developers and organizations building applications that require real-time, controllable speech generation, music generation, audio transcription, and omni-chat capabilities.

Highlights

  • Broad Model Support: Serves omni-chat models (Qwen3-Omni), music generation (MiniMax Music 3), TTS (Higgs Audio v3, MOSS-TTS), and ASR (Qwen3-ASR).
  • OpenAI-Compatible API: Provides endpoints for multimodal chat, streaming speech, and audio transcriptions.
  • Multi-Backend Hardware Support: Supports NVIDIA CUDA, Apple Silicon (experimental), and Intel GPU (experimental).
  • SGLang-Omni Router: Includes a multi-worker front door for health checks, capability discovery, and lifecycle management.

Related

  • Project
    sgl-project/sglang

    A high-performance serving framework for LLMs and multimodal models that provides low-latency, high-throughput inference across single GPUs and large distributed clusters.

  • Project
    vllm-project/vllm-omni

    A framework that extends vLLM to provide high-performance serving for omni-modality models, supporting text, image, audio, video, and action outputs across various hardware backends.

  • Project
    OpenMOSS/MOSS-TTS

    A family of open-source speech and sound generation models providing high-fidelity TTS, expressive dialogue, voice design, real-time interaction, and sound effect generation.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.

  • Project
    microsoft/onnxruntime-genai

    A high-performance API for running generative AI models on-device using ONNX Runtime, handling the complete generative loop from pre-processing to sampling.