sgl-project/sglang

What it solves

SGLang is a high-performance serving framework designed to solve the challenges of low-latency and high-throughput inference for large language models (LLMs) and multimodal models. It enables efficient deployment across various hardware setups, from single GPUs to large distributed clusters.

How it works

SGLang utilizes a fast runtime with several advanced optimization techniques to maximize performance:

• RadixAttention: Used for efficient prefix caching.
• Scheduling and Batching: Employs a zero-overhead CPU scheduler and continuous batching.
• Parallelism: Supports tensor, pipeline, expert, and data parallelism to handle large-scale workloads.
• Memory Management: Uses paged attention and chunked prefill.
• Decoding Optimizations: Implements speculative decoding and structured outputs.
• Quantization: Supports various formats including FP4, FP8, INT4, AWQ, and GPTQ.

Who it’s for

It is intended for developers and organizations deploying LLMs and multimodal models at scale, as well as researchers using it as a rollout backend for RL and post-training frameworks (such as AReaL, Miles, and verl).

Highlights

• Broad Model Support: Compatible with Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, and Mistral, as well as embedding, reward, and diffusion models.
• ** uma Hardware Compatibility**: Runs on NVIDIA GPUs, AMD GPUs, Intel Xeon CPUs, Google TPUs, and Ascend NPUs.
• Industry Adoption: Powering over 400,000 GPUs worldwide and used by major tech companies like xAI, NVIDIA, and AMD.
• OpenAI API Compatibility: Compatible with most Hugging Face models and OpenAI APIs.