OpenPipe/ART

OpenPipe / ART – Agent Reinforcement Trainer

What it is – ART (Agent Reinforcement Trainer) is an open‑source Python library that lets you turn a large language model (LLM) into a reinforcement‑learning (RL) trained agent. It provides a thin, OpenAI‑compatible client that your existing code can call, and a separate server that runs the heavy‑weight inference and LoRA‑based RL updates (using the GRPO algorithm). The whole loop can be run on a local GPU or on a managed, server‑less backend backed by Weights & Biases (W&B) so you don’t have to provision clusters yourself.

---

Core ideas

Concept	What it means for you
GRPO	A policy‑gradient method (Generalized‑Reward‑Based Policy Optimization) that updates a LoRA adapter on top of a frozen base model.
Server‑less RL	ART can spin up a temporary GPU‑enabled environment in the cloud (via the W&B backend) and automatically handle scaling, checkpointing and inference serving.
Client‑server split	Your application talks to a lightweight client that looks like the OpenAI chat API. The server does the actual model inference (via vLLM) and the RL training.
Trajectories & rewards	Each rollout of your agent is recorded as a sequence of messages (system/user/assistant). After the rollout you assign a scalar reward, which the server uses to compute policy gradients.

---

Quick start (one‑liner)

from art.serverless.backend import ServerlessBackend
import art

model = art.TrainableModel(
    project="voice‑agent",
    name="agent‑001",
    run_name="agent‑001",
    base_model="Qwen/Qwen3.6-27B"
)
backend = ServerlessBackend(api_key="YOUR_WANDB_KEY")
model.register(backend)   # launches a managed GPU env
# now use model like an OpenAI chat client
response = model.chat([{"role": "user", "content": "Schedule a meeting tomorrow"}])

The code above creates a LoRA‑enabled model, registers it with the server‑less backend, and you can start sending prompts immediately. Behind the scenes ART will:

1. Run the prompt on the latest LoRA via vLLM.
2. Store the interaction as a trajectory.
3. When you later call model.train() (or let the client auto‑train after a batch of rollouts) the server updates the LoRA with GRPO and makes the new checkpoint instantly available.

---

Notable notebooks (hands‑on demos)

Task	Notebook	Model used	What you’ll see
Email‑search agent (ART·E)	train‑art‑e.ipynb	Qwen‑3.8 27B + RULER	Agent learns to retrieve relevant emails; training curve shown.
Play 2048	2048.ipynb	Qwen‑3.8 27B	RL‑trained agent improves its game score over thousands of rollouts.
Tic‑Tac‑Toe	tic‑tac‑toe.ipynb	Qwen‑2.5 3B	Simple turn‑based game; visual benchmark of win‑rate growth.
Codenames	Codenames_RL.ipynb	Qwen‑2.5 3B	Agent learns to give clues and guess words; win‑rate plotted.
SFT + RL summarizer	sft‑rl.ipynb	Qwen‑2.5 7B	Warm‑up with supervised fine‑tuning, then RL fine‑tuning for better summaries.

---

How the training loop works

1. Inference – Your app calls model.chat(...). The request is sent to the ART server, which runs the current LoRA on the base model with vLLM and returns the completion. All messages are stored in a trajectory.
2. Reward assignment – After a rollout finishes you attach a numeric reward (e.g., success/failure, score, or a reward generated by the RULER system).
3. Batch & send – Once enough trajectories are collected, the client batches them and sends them to the server.
4. GRPO update – The server computes policy gradients using the rewards, updates the LoRA adapter, writes a new checkpoint, and reloads it into the inference engine.
5. Loop repeats – Inference resumes with the freshly updated policy; the process continues until you hit a target number of iterations or a performance threshold.

---

Key benefits claimed by the authors

• Cost‑effective – Shared inference cluster reduces per‑run cost by ~40 %.
• Speed – Ability to handle > 2000 concurrent requests, giving ~28 % faster wall‑clock training compared to a single‑GPU setup.
• Zero‑ops – The server‑less backend provisions GPU nodes, monitors health, and tears them down automatically.
• Instant checkpoint access – Every LoRA checkpoint is immediately usable for inference via the same client.
• Plug‑and‑play – Works with any model that runs in vLLM/HuggingFace‑transformers (most causal LLMs). The README notes Gemma 3 is currently unsupported.

---

Installation & basic usage

pip install openpipe-art   # pulls the library and its vLLM/GRPO deps

Then import the client (art.TrainableModel) and optionally a backend (art.serverless.backend.ServerlessBackend for W&B, or art.local.backend.LocalBackend for a local GPU).

Ecosystem integrations

• W&B – for managed server‑less RL, experiment tracking, and inference deployment.
• LangGraph – recent integration lets you train LangGraph‑based agent graphs with RL.
• Langfuse – observability of prompts, responses, and rewards.
• OpenPipe – the company behind ART provides additional hosted services (not required for the open‑source library).

---

Who might use ART?

• Research labs building multi‑step LLM agents (email assistants, tool‑using bots, game‑playing agents) that need RL fine‑tuning.
• Product teams that already have an LLM‑driven workflow and want to improve it via experience‑based learning without building their own RL infra.
• Educators / students looking for a ready‑made RL‑for‑LLMs sandbox; the Colab notebooks give a one‑click start.

---

License & citation

• License: Apache‑2.0 (permissive, commercial‑friendly).
• Citation: Provided BibTeX entry for academic use.

---

Bottom line

ART is a purpose‑built RL framework that abstracts away the heavy lifting of LoRA‑based policy optimization for LLM agents. By separating a lightweight client from a scalable server (optionally server‑less via W&B), it lets developers focus on defining environments, rollouts, and reward functions while the library handles inference, trajectory logging, and GRPO updates. If you need to turn an LLM into a self‑improving agent and want to avoid building RL pipelines from scratch, ART is a concrete, production‑ready option.