facebookresearch/LayerSkip

Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024

What it solves

LayerSkip addresses the computational cost and latency of Large Language Model (LLM) inference. It enables faster token generation by allowing the model to "exit early" from its deep stack of layers during the draft stage of generation, reducing the number of calculations required for each token.

How it works

LayerSkip uses a training recipe that allows a single model to act as both a draft model and a verification model. It implements self-speculative decoding, where the model exits at a specified layer (the exit_layer) to quickly generate draft tokens. These tokens are then verified by the full model in a single pass, significantly speeding up the process compared to standard autoregressive decoding.

Who it’s for

This project is for AI researchers and developers who want to increase the inference speed of Llama-based models without sacrificing correctness, particularly for generation tasks like summarization and coding.

Highlights

  • Self-Speculative Decoding: Uses the same model for both drafting and verification, eliminating the need for a separate, smaller draft model.
  • Early Exit Inference: Allows users to specify which layer to exit at to balance speed and accuracy.
  • Broad Model Support: Provides pre-trained checkpoints for various Llama 2, Llama 3, Llama 3.2, and CodeLlama models.
  • Integration: Integrated into Hugging Face transformers and PyTorch torchtune.
  • Evaluation Tools: Includes scripts for benchmarking, correctness verification, and hyperparameter sweeping to find the optimal exit layer and number of speculations.

Related

  • Dispatch
    Faster Text Generation with Self-Speculative Decoding

    Hugging Face introduces self-speculative decoding via LayerSkip, a method that uses a single LLM's early layers for drafting and later layers for verification to increase generation speed and reduce memory overhead.

  • Project
    vllm-project/speculators

    A library for training speculative decoding draft models that integrate directly with vLLM to reduce LLM inference latency without losing output quality.

  • Project
    z-lab/dflash

    A lightweight block diffusion model for speculative decoding that enables efficient, high-quality parallel drafting to speed up LLM generation.

  • Dispatch
    DeepSeek DSpark: Accelerating LLM Inference by 60-85%

    DeepSeek has open-sourced DSpark, an inference optimization technique that achieves 60-85% faster generation speeds by improving upon speculative decoding.

  • Project
    sgl-project/SpecForge

    A training framework for speculative decoding models designed for seamless integration with SGLang to accelerate LLM inference.