huggingface/alignment-handbook

What it solves

The Alignment Handbook provides a standardized, robust set of training recipes to help the community align language models with human and AI preferences. It fills the gap in public resources for training chat models, specifically focusing on how to collect data and use advanced alignment techniques beyond simple supervised fine-tuning (SFT).

How it works

The project is organized into two main components:

• Scripts: Training and evaluation scripts that support distributed training (via DeepSpeed ZeRO-3) or parameter-efficient fine-tuning (LoRA/QLoRA). These scripts cover the full pipeline: continued pretraining, SFT, and preference alignment using methods like DPO and ORPO.
• Recipes: YAML configuration files that contain the exact parameters for specific training runs, allowing users to reproduce models like Zephyr 7B or SmolLM.

Who it’s for

It is designed for machine learning practitioners and researchers who want to train, adapt, or align open-source LLMs to follow instructions and adhere to specific preferences.

Highlights

• Comprehensive Pipeline: Supports continued pretraining, SFT, reward modeling, rejection sampling, and preference optimization.
• Advanced Alignment Techniques: Implements Direct Preference Optimization (DPO), Odds Ratio Preference Optimization (ORPO), and Constitutional AI.
• Reproducibility: Provides specific recipes to replicate state-of-the-art small LLMs and coding assistants.
• Flexible Training: Supports full weight training with DeepSpeed or efficient tuning via LoRA/QLoRA.