SGLang Transformers Backend Integration

SGLang has integrated the Hugging Face transformers library as a backend, allowing users to run any transformers-compatible model with high-performance inference capabilities. This integration bridges the gap between the flexibility of the transformers ecosystem and the the high-throughput, low-latency requirements of production environments.

High-Performance Inference via SGLang

SGLang is designed for efficiency, utilizing features such as RadixAttention—a memory-efficient attention mechanism—to provide faster and more resource-efficient inference, particularly under heavy load. While the standard transformers library is optimized for experimentation, small-scale tasks, and training, SGLang provides a specialized engine for high-volume scenarios.

SGLang offers multiple deployment modes:

  • Offline Engine: Using sgl.Engine for direct generation.
  • Server Mode: Launching a server via sglang.launch_server to handle requests via an API.
  • OpenAI-Compatible API: Providing a drop-in replacement for external services.

Transformers Backend Integration

The transformers backend allows SGLang to automatically fall back to the transformers library for models that are not natively supported by SGLang. This provides several immediate benefits:

  • Instant Access: Users can run new models added to the transformers library immediately.
  • Custom Model Support: Models hosted on the Hugging Face Hub that are compatible with transformers can be deployed.
  • Reduced Engineering Overhead: Users no longer need to wait for native SGLang implementation for every new model architecture.

This integration enables optimized deployment and the use of RadixAttention for a wider range of models without sacrificing the versatility of the transformers ecosystem.

Usage and Compatibility

To use the transformers backend, users can explicitly set the implementation parameter in the engine initialization:

llm = sgl.Engine(model_path="meta-llama/Llama-3.2-1B-Instruct", impl="transformers")

Specifying impl="transformers" is optional; SGLang will automatically switch to the transformers implementation if the model is not natively supported.

Compatibility Requirements

Any model on the Hugging Face Hub that meets the following criteria is compatible:

  1. It works with the transformers library.
  2. It properly implements attention.
  3. For custom models, trust_remote_code=True is set during loading.

For example, the Kyutai Team’s Helium model (kyutai/helium-1-preview-2b), which is not yet natively supported by SGLang, can be run using the transformers backend:

python3 -m sglang.launch_server \
  --model-path kyutai/helium-1-preview-2b \
  --impl transformers \
  --host 0.0.0.0 \
  --port 30000

Future Roadmap

Development is ongoing to improve the integration in the following areas:

  • Performance Optimization: Reducing the performance gap between transformers-backend models and those with native SGLang integration.
  • LoRA Support: Adding support for Low-Rank Adaptation.
  • VLM Integration: Extending support to Vision-Language Models (VLM) to expand use cases.

Sources