chrisliu298/awesome-on-policy-distillation

A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation (OPD) of large language models

What it solves

This project provides a curated collection of resources for On-Policy Distillation (OPD), a technique used to train smaller "student" large language models (LLMs) using a larger "teacher" model. Unlike traditional distillation, OPD addresses the "train-inference distribution gap" by training the student on samples it generates itself, rather than on fixed data traces, ensuring the student learns how to correct its own mistakes.

How it works

In an OPD loop, the student model generates trajectories (rollouts). A teacher model then provides dense, token-level supervision or scoring on those specific student-generated samples. This process can be implemented in several ways:

  • White-box: The student has access to the teacher's internal probability distributions (logits).
  • Black-box: The teacher is accessed via API, and the student learns from rewards or discriminators.
  • Self-distillation: The model acts as both teacher and student, often using privileged context to improve itself.

Who it’s for

  • AI Researchers: Those studying model compression, knowledge distillation, and post-training alignment.
  • ML Engineers: Developers looking to implement industrial-grade distillation recipes (like those used by Qwen, DeepSeek, or NVIDIA) to create efficient, high-performance small models.
  • LLM Practitioners: Anyone wanting to reduce the gap between how a model is trained and how it actually performs during inference.

Highlights

  • Comprehensive Taxonomy: Organizes hundreds of papers and reports by their role in the OPD loop (foundations, gap-bridging, stability, etc.).
  • Industrial Recipes: Includes technical reports from major labs like Alibaba (Qwen3), DeepSeek (V4), and NVIDIA (Nemotron-Cascade 2).
  • Implementation Guides: Links to frameworks and tools like TRL's DistillationTrainer for accelerated training.
  • Diverse Modalities: Covers extensions of OPD into agents, multimodal models, speech, and robotics.

Related

  • Project
    vvvvvjdy/D-OPSD

    D-OPSD is an on-policy self-distillation framework that allows step-distilled diffusion models to be continuously tuned for new styles and concepts while preserving their fast inference capabilities.

  • Project
    MeiGen-AI/OPSD-V

    An on-policy self-distillation framework for post-training few-step autoregressive video generators to reduce error accumulation and improve motion dynamics in long videos.

  • Dispatch
    Self-Distillation: A New Frontier for Continual Learning in LLMs

    Exploring how self-distillation transforms off-policy data into on-policy data to mitigate catastrophic forgetting during LLM fine-tuning.

  • Project
    worldbench/DanceOPD

    DanceOPD is a research library that distills multiple diffusion‑based image‑generation capabilities (text‑to‑image, local edit, global edit, realism, etc.) into one model. It treats each teacher as a velocity field, queries the teacher on the student’s own rollout states, and trains with a simple velocity‑MSE loss. The repo supports SD‑3.5‑Medium and Z‑Image backends, includes a tiny smoke‑test, and provides scripts/configs for full training with LoRA adapters.