sgl-project/SpecForge

Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

What it solves

SpecForge addresses the lack of well-maintained, compatible training frameworks for speculative decoding models. It provides a standardized way to train these models so they can be seamlessly ported to the SGLang serving framework to accelerate LLM inference speed.

How it works

SpecForge uses a unified, typed training entry point that allows users to configure different drafting methods and deployment topologies via YAML files. It supports multiple training modes, including online disaggregated training and both colocated and disaggregated offline training. The framework supports various data, tensor, and sequence parallel topologies to optimize the training process.

Who it’s for

It is designed for developers and researchers who want to train speculative decoding draft models to improve the inference performance of their LLMs, specifically those using the SGLang serving framework.

Highlights

  • SGLang Compatibility: Models trained with SpecForge require no additional porting effort to work with SGLang.
  • Multiple Drafting Methods: Supports a wide array of techniques including EAGLE3, P-EAGLE, DFlash, DFlash2, Domino, and DSpark.
  • Flexible Training Topologies: Supports online and offline training with options for colocated or disaggregated setups.
  • SpecBundle: Provides a collection of production-grade pre-trained speculative decoding models for immediate use.

Related

  • Project
    vllm-project/speculators

    A library for training speculative decoding draft models that integrate directly with vLLM to reduce LLM inference latency without losing output quality.

  • Project
    sgl-project/sglang

    A high-performance serving framework for LLMs and multimodal models that provides low-latency, high-throughput inference across single GPUs and large distributed clusters.

  • Project
    baidu-baige/LoongForge

    A high-performance training framework for LLMs, VLMs, diffusion, and embodied models that optimizes parallelism and kernel efficiency to reduce training costs.

  • Project
    THUDM/slime

    An LLM post-training framework for RL scaling that integrates Megatron and SGLang to provide high-performance training and flexible data generation.

  • Dispatch
    vLLM Speculators: Parallel Drafting for Speculative Decoding

    vLLM and the Speculators project introduce open-source support for P-EAGLE, DFlash, and DSpark, moving beyond autoregressive drafting to generate candidate token blocks in parallel for faster LLM inference.