allenai/open-instruct

AllenAI's post-training codebase

What it solves

Open Instruct provides a unified framework for instruction-tuning and post-training popular pretrained language models using publicly available datasets. It aims to make the process of creating high-quality, instruction-following models (like the Tülu series) open and reproducible by providing the necessary code, recipes, and artifacts.

How it works

The project implements a multi-stage post-training pipeline:

  • Supervised Fine-Tuning (SFT): Uses a unified format for instruction datasets to finetune base models.
  • Preference Tuning: Implements Direct Preference Optimization (DPO) and PPO to align models with human preferences.
  • Reinforcement Learning with Verifiable Rewards (RLVR): Employs techniques like GRPO to train models using rewards that can be verified.
  • Parameter-Efficient Tuning: Supports LoRA and QLoRA for more accessible finetuning.

Who it’s for

Researchers and developers who want to reproduce the Tulu models or apply advanced post-training techniques (SFT, DPO, RLVR) to open-source language models.

Highlights

  • Comprehensive Pipeline: Covers the full spectrum from SFT to preference tuning and RLVR.
  • Tülu Model Suite: Provides the training recipes and checkpoints for the Tülu 1, 2, and 3 series of models.
  • Decontamination Tools: Includes scripts to measure overlap between training and evaluation datasets to ensure fair assessment.
  • Broad Model Support: Compatible with Llama 3.1 and OLMo 2 models.

Related

  • Dispatch
    Fine-tuning Llama 2 with Direct Preference Optimization (DPO) via TRL

    Hugging Face has integrated Direct Preference Optimization (DPO) into the TRL library, enabling the alignment of Llama 2 and other LLMs using a simplified binary cross-entropy loss instead of complex RLHF pipelines.

  • Project
    huggingface/trl

    A library for post-training foundation models using techniques like SFT, DPO, and GRPO to align them with human preferences and specific tasks.

  • Project
    SalesforceAIResearch/DiffusionDPO

    A training framework for Diffusion-DPO that aligns image generation models like Stable Diffusion with human preferences using Direct Preference Optimization.

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is an open‑source Python framework that applies a unified reinforcement‑learning loop to many multimodal generative models (LLMs, vision‑language models, image/video diffusion, and hybrid AR‑diffusion models). It provides four entry points, Hydra‑based configs, pluggable rollout engines, and distributed training via Ray and FSDP. The repo ships standard RL algorithms plus three novel team‑proposed methods (Flow‑DPPO, DRPO, CPPO) with tutorials. Supported models include Stable Diffusion 3, FLUX.2‑Klein, Qwen‑3, HunyuanVideo, and more. Documentation, example recipes, and a WeChat community are provided. Licensed under Apache‑2.0.

  • Project
    Goekdeniz-Guelmez/mlx-lm-lora

    A toolkit for training and fine-tuning Large Language Models locally on Apple Silicon using MLX, supporting a wide range of preference optimization and quantization-aware training methods.