huggingface/trl

Train transformer language models with reinforcement learning.

TL;DR

TRL (Transformers Reinforcement Learning) is a Hugging Face library that adds a collection of ready‑to‑use trainer classes for post‑training large language models. It implements several modern alignment/fine‑tuning algorithms (SFT, GRPO, DPO, KTO, Reward, Distillation) and plugs into the 🤗 Transformers, Accelerate, PEFT, and DeepSpeed ecosystems, so you can run on a single GPU or a multi‑node cluster with minimal code.

What it does

  • Supervised Fine‑Tuning (SFT) – classic instruction‑following fine‑tuning.
  • Group Relative Policy Optimization (GRPO) – a memory‑efficient alternative to PPO, used to train DeepSeek‑R1.
  • Direct Preference Optimization (DPO) – preference‑based alignment that powered Llama 3.
  • Kahneman‑Tversky Optimization (KTO) – learns from binary “good/bad” feedback.
  • Reward modeling – trains a reward model on preference data.
  • DistillationTrainer – on‑policy knowledge distillation with a chunked JSD loss, accelerated by vLLM.

All trainers are thin wrappers around the 🤗 Transformers Trainer, inheriting its support for distributed strategies (DDP, DeepSpeed ZeRO, FSDP) and mixed‑precision.

Key integrations

  • 🤗 Accelerate – easy scaling from one GPU to many nodes.
  • 🤗 PEFT – LoRA/QLoRA quantization lets you fine‑tune multi‑billion‑parameter models on modest hardware.
  • 🦥 Unsloth – optional optimized kernels for faster training.
  • CLI – trl sft, trl dpo, trl kto, etc., let you launch common jobs without writing Python code.

Quick start (Python)

from trl import SFTTrainer
from datasets import load_dataset

ds = load_dataset("trl-lib/Capybara", split="train")
trainer = SFTTrainer(model="Qwen/Qwen2.5-0.5B", train_dataset=ds)
trainer.train()

Replace SFTTrainer with GRPOTrainer, DPOTrainer, KTOTrainer, or RewardTrainer and adjust the model/dataset as shown in the README for other algorithms.

Quick start (CLI)

trl sft \
  --model_name_or_path Qwen/Qwen2.5-0.5B \
  --dataset_name trl-lib/Capybara \
  --output_dir ./qwen-sft

Similar commands exist for dpo and kto.

Installation

pip install trl               # stable release
# or, for the latest code
pip install git+https://github.com/huggingface/trl.git

Who should use it?

  • Researchers and engineers who need a battle‑tested implementation of recent LLM alignment algorithms.
  • Teams that already work with 🤗 Transformers and want to add RL‑style fine‑tuning without building the training loop from scratch.
  • Anyone looking for a CLI‑driven way to experiment with SFT, DPO, GRPO, KTO, or distillation on their own datasets.

Where to learn more

  • Official docs: https://huggingface.co/docs/trl
  • Papers linked in the README (e.g., GRPO 2024‑02‑03, DPO 2023‑05‑18).
  • The trl.experimental package for bleeding‑edge features.

Citation

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and others},
  year    = {2020},
  url     = {https://github.com/huggingface/trl},
  license = {Apache-2.0}
}

Related

  • Dispatch
    Fine-tuning Llama 2 with Direct Preference Optimization (DPO) via TRL

    Hugging Face has integrated Direct Preference Optimization (DPO) into the TRL library, enabling the alignment of Llama 2 and other LLMs using a simplified binary cross-entropy loss instead of complex RLHF pipelines.

  • Dispatch
    Vision Language Model Alignment in TRL

    Hugging Face has expanded the TRL library to support advanced alignment methods for Vision Language Models, including MPO, GRPO, and GSPO, alongside native SFT support and vLLM integration.

  • Dispatch
    Finetuning Stable Diffusion with DDPO via TRL

    Hugging Face has integrated Denoising Diffusion Policy Optimization (DDPO) into the TRL library, enabling the alignment of Stable Diffusion models with human preferences using reinforcement learning.

  • Dispatch
    Fine-tuning 20B LLMs with RLHF on a 24GB Consumer GPU

    Hugging Face released an integration of TRL and PEFT that enables reinforcement learning fine‑tuning of 20‑billion‑parameter language models on a single 24 GB GPU using 8‑bit quantization and low‑rank adapters.

  • Dispatch
    TRL v1.0 release notes / what's new

    Hugging Face releases TRL v1.0, a stable post-training library implementing over 75 methods, featuring a dual-track stability model to balance rapid experimental iteration with production-grade reliability.