Fine-tuning Llama 2 with Direct Preference Optimization (DPO) via TRL

Hugging Face has integrated Direct Preference Optimization (DPO) into the TRL library, allowing developers to align Large Language Models (LLMs) like Llama 2 using a simplified binary cross-entropy loss. This method removes the need for an auxiliary reward model and the complex reinforcement learning (RL) machinery typically required for Reinforcement Learning from Human Feedback (RLHF).

DPO vs. PPO: Simplifying the Alignment Pipeline

Direct Preference Optimization (DPO) simplifies the alignment process by bypassing the reward modeling and RL optimization steps. In traditional RLHF pipelines, such as those using Proximal Policy Optimization (PPO), a reward model is trained to estimate human preferences and a frozen reference model is used with a KL penalty to prevent the policy model from deviating too far and producing gibberish.

DPO replaces this with an analytical mapping from the reward function to the optimal RL policy. This allows the RL loss over reward and reference models to be transformed into a loss over the reference model directly. Consequently, the language model is optimized using a direct likelihood objective on preference data, eliminating the need for a separate reward model or fiddly RL-based optimization.

Implementing DPO with the TRL Library

The TRL library provides helpers to implement the DPO workflow. While a traditional RLHF pipeline involves four steps—Supervised Fine-Tuning (SFT), preference data annotation, reward model training, and RL optimization—DPO collapses the latter two into a single step.

Data Requirements

To use the DPOTrainer in TRL, preference data must be provided in a dictionary format containing three specific keys:

  • prompt: The context prompt provided to the model at inference time.
  • chosen: The preferred generated response.
  • rejected: The non-preferred response.

Trainer Configuration

The DPOTrainer requires a base model (from the SFT pipeline), a reference model (typically a copy of the SFT-trained base model), and a temperature hyperparameter beta (typically between 0.1 and 0.5). The beta parameter controls the influence of the reference model; a smaller beta value means the reference model is more ignored.

Case Study: Aligning Llama 2 7B

Hugging Face demonstrated the application of DPO by fine-tuning a Llama 2 7B-parameter model on the stack-exchange preference dataset, which contains ranked answers to questions.

Step 1: Supervised Fine-Tuning (SFT)

The model first undergoes SFT using the SFTTrainer and the QLoRA technique via the bitsandbytes library. This involves loading the base model in 4-bit quantization and adding LoRA layers to the target modules (q_proj, v_proj).

Step 2: DPO Training

Following SFT, the resulting model is used as both the base model and the reference model for DPO training. Using AutoPeftModelForCausalLM, the models are loaded in 4-bit configuration and trained via the QLoRA method. The DPOTrainer evaluates progress using an evaluation dataset and reports implicit reward metrics.

Monitoring DPO Performance

During training and evaluation, the DPOTrainer tracks four key reward metrics to measure alignment progress:

  • rewards/chosen: The mean difference between the log probabilities of the policy model and the reference model for chosen responses, scaled by beta.
  • rewards/rejected: The mean difference between the log probabilities of the policy model and the reference model for rejected responses, scaled by beta.
  • rewards/accuracies: The percentage of times the chosen rewards are higher than the corresponding rejected rewards.
  • rewards/margins: The mean difference between the chosen and rejected rewards.

Successful training is indicated by increasing margins and accuracies approaching 1.0, meaning the model consistently assigns higher rewards to preferred responses.

Sources