ZJU-REAL/SDAR

📚 What is SDAR?

Self‑Distilled Agentic Reinforcement Learning (SDAR) is an open‑source research framework that lets you train agentic reinforcement‑learning (RL) agents – i.e., LLM‑driven bots that act in interactive environments – using a self‑distillation paradigm. The core idea is to let a “teacher” model generate trajectories, then have a “student” model learn from those trajectories while a gating mechanism decides when to keep the teacher’s knowledge and when to let the student explore on its own. The repository ships the original SDAR algorithm plus a newer variant called RetireOPD (self‑retiring on‑policy distillation).

---

🚀 Key Features (as listed in the README)

Category	What the code supports
Method	OP(S)D, GRPO, GRPO+OP(S)D, RLSD, Skill‑SD, SDAR, RetireOPD
Environments	ALFWorld (text‑based household tasks), WebShop (e‑commerce interaction), Search‑QA
Model Back‑ends	Qwen‑3 series, Qwen‑2.5 series (LLM checkpoints)

Highlights

• Gating mechanism that decides when to apply self‑distillation vs. direct RL updates.
• Two‑stage training for RetireOPD: first train a teacher, then retire (phase‑out) the teacher while the student continues learning.
• Extensive ecosystem – the repo has already powered dozens of follow‑up papers (e.g., AlignOPSD, PR‑OPD, UniOPSD, etc.).
• Results on three benchmarks show > 90 % success on ALFWorld and strong performance on WebShop.

---

🛠️ Getting Started

1. Create a Python 3.12 environment and install the core dependencies (vllm, flash‑attn, the package itself).
2. Install the target environment you want to train on:
   • ALFWorld – gymnasium, stable‑baselines3, alfworld; then run alfworld-download -f.
   • WebShop – requires Python 3.10; follow the setup.sh script in agent_system/environments/.../webshop.
   • Search‑QA – install a small set of packages and set up a separate retriever conda env with faiss‑gpu, fastapi, etc.
3. Training scripts live under examples/. Example commands:

   bash examples/sdar_trainer/run_alfworld_3b.sh   # SDAR on ALFWorld with a 3‑B Qwen model
   bash examples/retireopd_trainer/run_alfworld_3b.sh   # RetireOPD (two‑stage)
   

4. Checkpoint handling – use scripts/model_merger.py to merge FSDP/Megatron shards into a single Hugging‑Face‑compatible checkpoint.

---

📊 Reported Results

Algorithm	Task	Model	Success Rate
RetireOPD	ALFWorld	Qwen2.5‑1.5B‑Instruct	89.8 %
RetireOPD	ALFWorld	Qwen2.5‑3B‑Instruct	93.8 %
RetireOPD	ALFWorld	Qwen2.5‑7B‑Instruct	95.3 %
RetireOPD	WebShop	Qwen2.5‑1.5B‑Instruct	75.8 %
RetireOPD	WebShop	Qwen2.5‑3B‑Instruct	77.3 %
RetireOPD	WebShop	Qwen2.5‑7B‑Instruct	84.4 %

---

📚 How to Cite

If you use SDAR or any of its derivatives, the README provides a ready‑to‑copy BibTeX entry:

@article{lu2026sdar,
  title={Self‑distilled agentic reinforcement learning},
  author={Lu, Zhengxi and …},
  journal={arXiv preprint arXiv:2605.15155},
  year={2026}
}

(Additional citations for related works like Skill0, AgentOPSD, etc., are also listed.)

---

🎯 Who Might Find This Useful?

• Researchers exploring LLM‑driven agents that need to learn from their own experience.
• Engineers building RL pipelines for text‑based or web‑based tasks and wanting a ready‑made self‑distillation implementation.
• Students looking for a concrete codebase that ties together recent papers on on‑policy distillation.

---

🔗 Related Projects

The README lists a long “Awesome Work Powered by SDAR” section, showing that many recent papers (e.g., AlignOPSD, PR‑OPD, UniOPSD) fork or extend this repository. Those projects can be explored for specialized variants.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.