aiming-lab/SkillRL

What it solves

SkillRL addresses the inefficiency of traditional memory-based LLM agents that store raw, noisy, and redundant trajectories. Instead of relying on raw experience, it provides a way to abstract these experiences into a hierarchical library of reusable behavioral patterns (skills), reducing token usage while improving reasoning utility.

How it works

The framework uses a recursive process to evolve an agent's policy through Reinforcement Learning (RL) and skill discovery:

1. Skill Distillation: It transforms successful trajectories into strategic patterns and failed ones into lessons on how to avoid mistakes.
2. Hierarchical SkillBank: Knowledge is organized into "General Skills" (universal strategies) and "Task-Specific Skills" (heuristics for specific categories).
3. Retrieval: During execution, the agent retrieves relevant skills using either keyword-based template matching or semantic similarity via an embedding model (e.g., Qwen3-Embedding).
4. Recursive Evolution: The skill library is dynamically updated during RL training by analyzing validation failures to refine or add new skills.

Who it’s for

It is designed for researchers and developers building LLM-based agents for complex environment tasks (such as ALFWorld, WebShop, and Search) who want to improve agent performance and efficiency using RL.

Highlights

• Token Compression: Reduces context usage by 10-20% compared to raw trajectory storage.
• Dynamic Skill Evolution: The skill library co-evolves with the agent's policy during training.
• Flexible Retrieval: Supports both simple template-based matching and advanced embedding-based semantic search.
• Comprehensive Training Pipeline: Includes stages for memory data generation, Supervised Fine-Tuning (SFT), and RL with SkillBank.