sgl-project/SpecForge
Train speculative decoding models effortlessly and port them smoothly to SGLang serving.
What it solves
SpecForge is designed to address the lack of well-maintained, SGLang-compatible training frameworks for speculative decoding models. It provides a unified, runnable out-of-the-box ecosystem to train draft models that can be seamlessly ported to the SGLang serving framework to accelerate LLM inference speed.
How it works
SpecForge uses a single, typed training entry point (specforge train) and configuration files to manage different training topologies. It supports various drafting methods including feature-based autoregressive drafting (EAGLE3, EAGLE3.1, P-EAGLE) and block-parallel drafting (DFlash, Domino, DSpark). The framework supports local offline training, server-only online-disaggregated training, and various parallel topologies (data, tensor, and sequence parallel).
Who it’s for
Developers and researchers who want to train their own speculative decoding draft models and deploy them using SGLang for high-performance inference.
Highlights
- SGLang Compatibility: Directly compatible with SGLang, requiring no additional effort for porting.
- Unified Training Entry Point: All methods use the same command-line interface for training.
- Diverse Drafting Methods: Supports a wide range of state-of-the-art drafting techniques like EAGLE3 and DFlash.
- Flexible Training Topologies: Supports offline, online, and disaggregated training workflows.
- SpecBundle: Provides a collection of production-grade draft models with high acceptance rates for immediate use.