sgl-project/sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

What it solves

SGLang provides a high-performance inference framework designed to speed up the serving of large language models (LLMs), vision-language models, and diffusion models. It specifically targets the bottlenecks of agentic workloads, large-scale serving, and reinforcement learning (RL) rollouts, where efficiency and throughput are critical.

How it works

SGLang operates as an optimized inference engine that supports a vast array of hardware platforms, including NVIDIA, AMD, Google TPU, Intel, Apple Silicon, and Huawei Ascend. It includes a built-in image and video generation engine (SGLang Diffusion) and integrates with various training frameworks for RL rollouts and orchestration tools for scaling and routing.

Who it’s for

This framework is intended for AI researchers, developers, and enterprises deploying production-grade LLMs and multimodal models who need high throughput and broad hardware compatibility.

Highlights

  • Broad hardware support across GPUs, TPUs, NPUs, and CPUs from multiple vendors.
  • Optimized for agentic workloads and RL rollouts.
  • Built-in support for image and video generation via SGLang Diffusion.
  • Integration with the wider SGL Ecosystem, including speculative decoding (SpecForge) and audio serving (SGLang Omni).

Related

  • Project
    ServerlessLLM/ServerlessLLM

    A high-performance system for deploying multiple AI models on shared GPUs, featuring ultra-fast checkpoint loading and GPU multiplexing to enable efficient serverless inference.

  • Project
    sgl-project/SpecForge

    SpecForge is a unified training framework for speculative decoding models that integrates directly with the SGLang serving engine to accelerate LLM inference.

  • Project
    sgl-project/sglang-omni

    A multi-stage serving runtime for omni, speech, and TTS models that optimizes generation across heterogeneous compute stages with an OpenAI-compatible API.

  • Project
    sgl-project/sglang-jax

    A high-performance JAX-based inference engine optimized for Google TPUs, providing high throughput and low latency for LLMs and multimodal models.

  • Dispatch
    SGLang Transformers Backend Integration

    SGLang now supports Hugging Face transformers as a backend, enabling high-performance inference for any transformers-compatible model without requiring native SGLang support.