huggingface/trl

Train transformer language models with reinforcement learning.

What it solves

TRL provides a comprehensive toolkit for the post-training of foundation models. It simplifies the process of aligning models to human preferences or specific tasks using reinforcement learning and fine-tuning techniques, allowing developers to move beyond initial pre-training to create more useful and safe models.

How it works

Built on the Hugging Face Transformers ecosystem, TRL provides specialized trainer classes that wrap the standard Transformers trainer. It implements several advanced post-training algorithms:

  • SFT (Supervised Fine-Tuning): Basic fine-tuning on a labeled dataset.
  • GRPO (Group Relative Policy Optimization): A memory-efficient algorithm used for reasoning models (like DeepSeek-R1).
  • DPO (Direct Preference Optimization): Aligns models based on paired preferences.
  • KTO (Kahneman-Tversky Optimization): Aligns models using binary feedback (desirable vs. undesirable) rather than pairs.

It integrates with Accelerate for multi-GPU scaling, PEFT for memory-efficient training (LoRA/QLoRA), and Unsloth for kernel-level acceleration.

Who it’s for

Machine learning engineers and researchers who need to post-train, align, or fine-tune large language models and foundation models across various hardware setups, from single GPUs to multi-node clusters.

Highlights

  • Diverse Trainer Suite: Includes dedicated trainers for SFT, GRPO, DPO, KTO, and Reward training.
  • Memory Efficiency: Uses a fused LM head to reduce peak VRAM usage by 52% to 82% for sequences up to 8k tokens.
  • Scalable Infrastructure: Native support for DDP, DeepSpeed ZeRO, and FSDP via Hugging Face Accelerate.
  • No-Code Option: Includes a Command Line Interface (CLI) for running post-training methods without writing Python code.

Related

  • Project
    NVIDIA-NeMo/RL

    A scalable post-training library for multimodal models that streamlines reinforcement learning and alignment using high-performance backends like Megatron Core and vLLM.

  • Project
    allenai/open-instruct

    A framework for instruction-tuning and post-training open-source language models using SFT, DPO, and reinforcement learning with verifiable rewards.

  • Project
    verl-project/verl

    A flexible and efficient RL training library for LLMs that simplifies the implementation of RLHF algorithms and scales to trillion-parameter models.

  • Project
    huggingface/alignment-handbook

    A collection of robust training recipes and scripts to continue pretraining and align language models with human and AI preferences using techniques like DPO and ORPO.

  • Dispatch
    Vision Language Model Alignment in TRL

    Hugging Face has expanded the TRL library to support advanced alignment methods for Vision Language Models, including MPO, GRPO, and GSPO, alongside native SFT support and vLLM integration.