NovaSky-AI/SkyRL

SkyRL: A Modular Full-stack RL Library for LLMs

What it solves

SkyRL is a modular, full-stack reinforcement learning (RL) library designed specifically for Large Language Models (LLMs). It addresses the challenge of training agents that can handle multi-turn, long-horizon tasks in real-world environments, such as software engineering (SWE-Bench) or complex knowledge work, where simple supervised learning is often insufficient.

How it works

SkyRL provides a comprehensive ecosystem of components to facilitate RL training:

  • skyrl: A unified library for RL training on local hardware, incorporating a modular training framework (skyrl-train) and a cross-platform backend for the Tinker API (skyrl-tx).
  • skyrl-agent: An agent layer dedicated to optimizing and scaling pipelines for multi-turn tool-use LLMs on long-horizon tasks.
  • skyrl-gym: A collection of RL environments (gymnasiums) for LLMs, featuring tasks in math, coding, search, and SQL, implemented using the Gymnasium API.

Who it’s for

It is intended for researchers and developers building frontier AI agents that require reinforcement learning to improve their performance on complex, multi-step tool-use tasks and real-environment interactions.

Highlights

  • Full-stack RL capability: Combines training frameworks, agent layers, and specialized environments.
  • Tinker API support: Allows users to run training scripts written for the Tinker API on their own GPUs.
  • Specialized Environments: Includes skyrl-gym for tool-use tasks like SQL and coding.
  • Proven Performance: Used to create models like SkyRL-SQL-7B, which outperforms GPT-4o on Text-to-SQL tasks.
  • ** robotics/embodied intelligence integration**: Integrated with Harbor to train terminal-use agents.

Related

  • Project
    alibaba/ROLL

    ROLL is an open‑source library for reinforcement‑learning training of large language models at scale. It provides Ray‑based distributed execution, supports many RL algorithms (PPO, GRPO, Reinforce++, etc.), works with Megatron‑Core, vLLM, SGLang, and LoRA, and offers pipelines for RL‑based fine‑tuning (RLVR) and multi‑turn agentic RL. The project includes extensive docs, hardware‑agnostic support (NVIDIA, AMD, Ascend), and built‑in observability.

  • Project
    langfengQ/verl-agent

    An extension of veRL for training LLM agents via reinforcement learning, featuring a step-independent rollout mechanism to efficiently handle long-horizon, multi-turn tasks.

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.

  • Project
    MushroomRL/mushroom-rl

    MushroomRL is a modular Python library for reinforcement learning that provides a wide array of classical and deep RL algorithms and integrates with popular tensor libraries and benchmarks.

  • Project
    AgileRL/AgileRL

    A Deep Reinforcement Learning library that uses evolutionary hyperparameter optimization to drastically reduce the time and cost of training RL models.