sgl-project/SpecForge

Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

What it solves

SpecForge addresses the fragmentation and maintenance challenges in the speculative decoding ecosystem. It provides a unified, production-ready framework for training speculative drafting models that are natively compatible with the SGLang serving engine, eliminating the need for complex porting efforts.

How it works

SpecForge uses a standardized, configuration-driven approach to train various speculative decoding methods. Users define their training topology—such as online disaggregated or offline colocated setups—via YAML configuration files. The framework manages the interaction between the trainer and the serving engine, supporting advanced parallelization strategies to ensure the resulting models can be immediately deployed for inference acceleration.

Who it’s for

This project is designed for AI engineers and researchers who need to train and deploy high-performance speculative decoding models for large language models. It is particularly useful for those already utilizing the SGLang ecosystem who want to improve inference throughput without managing disjointed or unmaintained training scripts.

Highlights

  • Directly compatible with SGLang for seamless deployment.
  • Supports multiple state-of-the-art methods including EAGLE3, P-EAGLE, DFlash, and DSpark.
  • Enables both online and offline training across various parallel topologies.
  • Includes access to SpecBundle, a collection of pre-trained, production-grade speculative models.

Related

  • Project
    sgl-project/sglang

    An open-source inference framework optimized for fast serving of LLMs, vision-language, and diffusion models, particularly for agentic workloads and RL rollouts.

  • Dispatch
    vLLM Speculators: Parallel Drafting for Speculative Decoding

    vLLM and the Speculators project introduce open-source support for P-EAGLE, DFlash, and DSpark, moving beyond autoregressive drafting to generate candidate token blocks in parallel for faster LLM inference.

  • Project
    vllm-project/speculators

    Speculators is a Python library that trains small “draft” models for speculative decoding, enabling faster, loss‑less LLM inference when deployed with vLLM.

  • Project
    baidu-baige/LoongForge

    LoongForge is Baidu’s open‑source training framework for LLMs, VLMs, diffusion and embodied models. It combines a patched Megatron‑LM stack with a torch‑native subsystem, adds MoE expert‑placement, FP8/INT4 support, long‑sequence pipelines and other speed‑ups, and ships ready‑made Docker images and launch scripts for dozens of popular models.

  • Project
    z-lab/dflash

    A lightweight block diffusion model for speculative decoding that enables efficient, high-quality parallel drafting to speed up LLM generation.