chrisliu298/awesome-on-policy-distillation

What it solves

This repository addresses the "train-inference distribution gap" (exposure bias) that occurs when Large Language Models (LLMs) are trained on fixed datasets (off-policy) but must generate text independently during inference. By focusing on on-policy distillation (OPD), it provides a curated collection of resources to help developers and researchers train student models on their own evolving outputs while receiving dense, token-level guidance from a teacher model.

How it works

On-policy distillation involves a loop where a student model generates trajectories (samples) from its own current policy. A teacher model—which may be an external, privileged, or even the same model (self-distillation)—then scores or provides supervision for those specific student-generated samples. This process ensures the student learns to correct its own mistakes and aligns its training distribution with its actual inference behavior.

Who it’s for

• AI Researchers studying post-training primitives, policy gradients, and knowledge distillation.
• ML Engineers implementing model compression or capability transfer for production LLMs.
• Developers looking for industrial recipes used by labs like Alibaba (Qwen), DeepSeek, and NVIDIA.

Highlights

• Comprehensive Taxonomy: Organizes hundreds of papers and tools by feedback signal, teacher access mode (white-box vs. black-box), and loss scope.
• Industrial Recipes: Includes technical reports from major labs (e.g., Qwen3, DeepSeek-V4, GLM-5) showing how OPD is used in production.
• Diverse Variants: Covers specialized techniques like self-distillation (no external teacher), black-box OPD (API-only teachers), and context internalization (distilling prompts into weights).
• Implementation Guides: Links to frameworks and engineering walkthroughs, including TRL's DistillationTrainer for high-speed distillation.