NovaSky-AI/SkyRL

SkyRL: A Modular Full-stack RL Library for LLMs

What it solves

SkyRL is a modular full-stack library designed to make reinforcement learning (RL) for Large Language Models (LLMs) more accessible and efficient. It specifically addresses the challenges of training agents that can handle long-horizon, real-world tasks involving multi-turn tool use, such as coding, SQL generation, and terminal interaction.

How it works

SkyRL is organized into several specialized components:

  • skyrl: A unified library for RL training on local hardware, combining a modular training framework (skyrl-train) and a cross-platform backend for the Tinker API (skyrl-tx).
  • skyrl-agent: An agent layer focused on optimizing and scaling pipelines for multi-turn tool use LLMs on long-horizon tasks.
  • skyrl-gym: A collection of RL environments implemented using the Gymnasium API, providing tasks for math, coding, search, and SQL.

Who it’s for

This library is designed for AI researchers and developers building RL-tuned LLMs, specifically those focusing on agentic workflows, tool-use capabilities, and long-horizon problem solving.

Highlights

  • Tinker API Support: Implements a backend for the Tinker API, allowing users to run training scripts written for that API on their own GPUs.
  • Modular Architecture: Provides a separate training framework, agent layer, and environment library (Gymnasium API).
  • Longe-Horizon Focus: Optimized for real-world tasks like SWE-Bench and terminal-use agents.
  • On-Policy Distillation: Supports advanced RL techniques such as on-policy distillation and fully async RL with in-flight weight updates.

Related

  • Project
    alibaba/ROLL

    An efficient scaling library for reinforcement learning with LLMs that optimizes training and inference across large GPU clusters using a distributed Ray-based architecture.

  • Project
    langfengQ/verl-agent

    An extension of veRL for training LLM agents via reinforcement learning, featuring a step-independent rollout mechanism to efficiently handle long-horizon, multi-turn tasks.

  • Project
    ServiceNow/PipelineRL

    A scalable asynchronous reinforcement learning framework for LLM agents that uses in-flight weight updates to maximize GPU utilization while maintaining on-policy data freshness.

  • Project
    MushroomRL/mushroom-rl

    MushroomRL is a modular Python library for reinforcement learning that provides a wide array of classical and deep RL algorithms and integrates with popular tensor libraries and benchmarks.

  • Project
    AgileRL/AgileRL

    AgileRL is a Python library for deep reinforcement learning that adds RLOps‑style tooling. It bundles many RL algorithms and provides evolutionary hyper‑parameter optimisation, population‑based distributed training, multi‑agent support, offline/bandit methods, and a cloud platform (Arena) for scaling experiments. Install via `pip install agilerl` (optional extras for Box2D, Arena, LLM fine‑tuning). Licensed under Apache‑2.0.