ZJU-REAL/SDAR

Official code for "Self-Distilled Agentic Reinforcement Learning"

What it solves

SDAR addresses the challenge of improving the performance of LLM-based agents in complex environments by unifying Agentic Reinforcement Learning (RL) with On-Policy Self-Distillation (OPSD). It provides a framework to enhance agent capabilities across various tasks, such as navigation, shopping, and search, where standard RL baselines often struggle.

How it works

SDAR implements a self-distilled agentic reinforcement learning method that utilizes a gating mechanism. It allows agents to learn from their own successful trajectories and evolve their skills through a process of self-distillation. The framework supports multiple RL methods including GRPO, RLSD, and Skill-SD, and is compatible with models like Qwen3 and Qwen2.5.

Who it’s for

This project is designed for AI researchers and developers working on agentic reinforcement learning, specifically those looking to implement or experiment with self-distillation techniques to improve LLM agent performance in interactive environments.

Highlights

  • Unified Framework: The first open-sourced framework that unifies Agentic RL with OP(S)D.
  • Broad Environment Support: Compatible with ALFWorld, WebShop, and Search-QA.
  • Broad Method Support: Supports various training methods including OPSD, GRPO, and the proprietary SDAR method.
  • Extensive Research Ecosystem: Serves as the foundation for numerous subsequent research works such as TASPO, AHEAD, and AgentOPSD.

Related

  • Project
    chrisliu298/awesome-on-policy-distillation

    A curated collection of papers, frameworks, and tools for on-policy distillation, a post-training technique that trains student LLMs on their own generated samples to bridge the train-inference distribution gap.

  • Dispatch
    Self-Distillation: A New Frontier for Continual Learning in LLMs

    Exploring how self-distillation transforms off-policy data into on-policy data to mitigate catastrophic forgetting during LLM fine-tuning.

  • Project
    Thinking-Space/Rethinking-OPD

    A research project and toolkit for analyzing and improving On-Policy Distillation (OPD) in LLMs, providing strategies to recover failing distillation and understanding its token-level mechanisms.

  • Project
    NovaSearch-Team/RAG-Retrieval

    A comprehensive toolkit for training, fine-tuning, and distilling RAG retrieval models, including embedding models, late interaction models, and rerankers.

  • Project
    vvvvvjdy/D-OPSD

    D-OPSD is an on-policy self-distillation framework that allows step-distilled diffusion models to be continuously tuned for new styles and concepts while preserving their fast inference capabilities.