thunlp/OPD

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

What it solves

This project addresses the lack of understanding regarding the training dynamics and failure modes of On-Policy Distillation (OPD) in Large Language Models. It provides a systematic investigation into why OPD succeeds or fails and offers practical strategies to recover performance when distillation fails.

How it works

OPD involves training a student model using token-level reward signals provided by a teacher model. The project identifies two critical success conditions: the student and teacher must share compatible thinking patterns, and the teacher must provide capabilities that the student has not already encountered.

To improve failing OPD, the project implements two specific strategies:

  1. Off-policy cold start: Initializing the process to bridge the gap between models.
  2. Teacher-aligned prompt selection: Choosing prompts that better align the student with the teacher's capabilities.

Technically, it uses the verl framework for RL and distillation, and LlamaFactory for supervised fine-tuning (SFT), allowing for various Top-K token selection and weighting strategies to optimize the reward signal.

Who it’s for

Researchers and developers working on model distillation, post-training of LLMs, and those looking to optimize the transfer of knowledge from large teacher models to smaller student models.

Highlights

  • Mechanistic Analysis: Provides token-level analysis showing that successful OPD relies on alignment of high-probability tokens at student-visited states.
  • Recovery Strategies: Introduces off-policy cold start and teacher-aligned prompt selection to fix failing distillation runs.
  • Integration: Merged diagnostic metrics (overlap_ratio and overlap_token_advantage) into the verl library.
  • Flexible Configuration: Supports multiple Top-K strategies (union, intersection, etc.) and reward weighting schemes.

Related

  • Project
    chrisliu298/awesome-on-policy-distillation

    A curated collection of papers, frameworks, and tools for on-policy distillation, a post-training technique that trains student LLMs on their own generated samples to bridge the train-inference distribution gap.

  • Project
    vvvvvjdy/D-OPSD

    D-OPSD is an on-policy self-distillation framework that allows step-distilled diffusion models to be continuously tuned for new styles and concepts while preserving their fast inference capabilities.

  • Dispatch
    Self-Distillation: A New Frontier for Continual Learning in LLMs

    Exploring how self-distillation transforms off-policy data into on-policy data to mitigate catastrophic forgetting during LLM fine-tuning.

  • Project
    MeiGen-AI/OPSD-V

    An on-policy self-distillation framework for post-training few-step autoregressive video generators to reduce error accumulation and improve long-horizon motion dynamics.

  • Project
    worldbench/DanceOPD

    DanceOPD is a distillation framework for flow-matching image generators that combines multiple capabilities like text-to-image and editing into one model without causing interference.