allenai/open-instruct

AllenAI's post-training codebase

What it solves

Training large language models to follow instructions and be helpful is a complex, fast-moving area. This project provides an open, unified codebase for instruction tuning and post-training of pretrained models using publicly available datasets. It lets researchers reproduce state-of-the-art results like the Tülu models without reinventing the training pipeline.

How it works

The repo contains training scripts for three main approaches:

  • Supervised fine-tuning (SFT) for instruction following.
  • Direct Preference Optimization (DPO) for learning from human preferences.
  • Reinforcement Learning with Verifiable Rewards (RLVR) via GRPO (a policy-gradient method), which is useful when a task has a clear, checkable reward (like code correctness).

Datasets are brought into a unified format. The codebase also includes decontamination scripts to measure overlap between instruction tuning and evaluation datasets, which helps avoid accidental data leakage. Launch scripts are provided for AI2's Beaker cluster, but training can also be run locally. The project notes that native evaluation support is now unmaintained and suggests using OLMES instead.

Who it’s for

This is aimed at researchers and developers who want to train or fine-tune language models on open data, especially those experimenting with instruction following, preference tuning, or reward-based reinforcement learning. It’s a research codebase, so it doesn’t guarantee backward compatibility.

Highlights

  • Provides ready-to-run scripts for SFT, DPO, and GRPO-based RLVR.
  • Includes decontamination tooling to prevent train/eval data overlap.
  • Based on the Tülu model series and used for OLMo 3 experiments.
  • Code is adapted from several well-known projects (HuggingFace, TRL, OpenAI RLHF, OpenRLHF).
  • Licensed under Apache 2.0.

Related

  • Project
  • Project
  • Project
  • Project
  • Dispatch