Thinking-Space/Rethinking-OPD

What it solves

This project investigates the dynamics and mechanisms of On-Policy Distillation (OPD) for Large Language Models (LLMs). It addresses the lack of understanding regarding why OPD sometimes fails and provides practical strategies to recover performance when distillation does not succeed.

How it works

The project analyzes the interaction between a student model and a teacher model during distillation. It identifies two critical success conditions: the student and teacher must share compatible thinking patterns, and the teacher must provide capabilities that the student has not already encountered during training.

To improve OPD, the project implements:

• Off-policy cold start: A strategy to initialize the student model.
• Teacher-aligned prompt selection: A method to choose prompts that better align with the teacher's strengths.
• Token-level reward signals: Using a teacher model to provide dense rewards based on token probabilities (with various Top-K selection and weighting strategies).

Who it’s for

Researchers and developers working on LLM post-training, specifically those using distillation to transfer knowledge from larger, more capable teacher models to smaller student models.

Highlights

• Mechanism Analysis: Proves that successful OPD relies on progressive alignment on high-probability tokens at states visited by the student.
• Practical Recovery: Offers specific recipes (cold start and prompt selection) to fix failing distillation runs.
• Integration: Includes diagnostics merged into the verl framework to track overlap ratios and token advantage.
• Flexible Training: Supports multiple Top-K strategies (union, intersection, etc.) and weighting schemes for token rewards.