Preference Tuning LLMs with Direct Preference Optimization Methods – Empirical Comparison of DPO, IPO, and KTO

TL;DR – Hugging Face evaluated three reinforcement‑learning‑free alignment methods—Direct Preference Optimization (DPO), Identity Preference Optimisation (IPO) and Kahneman‑Tversky Optimisation (KTO)—on two 7B chat models, finding DPO consistently outperforms the others when hyper‑parameters (especially the β weight) are carefully tuned.

Introduction

The blog post reports an empirical comparison of three recent LLM alignment algorithms that avoid traditional reinforcement learning. The study fine‑tunes two high‑quality 7B models (OpenHermes‑2.5‑Mistral‑7B and Zephyr‑7b‑beta‑sft) using three loss functions (DPO, IPO, KTO) across a sweep of the β hyper‑parameter and evaluates the resulting models with MT‑Bench, a GPT‑4‑based multi‑turn benchmark covering eight capability categories.

Alignment Methods Without Reinforcement Learning

  • Direct Preference Optimization (DPO) – Recasts alignment as a simple loss over paired preference tuples ((x, y_w, y_l)) and has been used to train models such as Zephyr and Intel’s NeuralChat.
  • Identity Preference Optimisation (IPO) – Extends DPO with a regularisation term to improve robustness and allow training to convergence without early stopping.
  • Kahneman‑Tversky Optimisation (KTO) – Uses only binary “good”/“bad” labels rather than paired preferences, enabling alignment from cheaper feedback signals like thumbs‑up/down.

All three methods are implemented in the 🤗 TRL library via the DPOTrainer with the loss_type argument set to sigmoid (DPO), ipo (IPO) or kto_pair (KTO).

Experimental Setup

  • Models: OpenHermes‑2.5‑Mistral‑7B (un‑aligned 7B chat model) and Zephyr‑7b‑beta‑sft (already fine‑tuned on supervised data).
  • Datasets:
    • orca_dpo_pairs (13 k prompt pairs, GPT‑4 chosen vs. Llama‑Chat‑13B rejected) – used for OpenHermes.
    • ultrafeedback‑binarized (66 k prompt pairs) – used for Zephyr.
  • Training configuration: One epoch, batch size 8 per device, learning rate 5e‑7, cosine scheduler, β values from 0.01 to 0.9, gradient checkpointing, bf16 enabled, and evaluation every 100 steps.
  • Evaluation: MT‑Bench, which uses GPT‑4 to score model responses across Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, and Humanities.

The full configuration files and scripts are available in the Hugging Face alignment‑handbook repository.

Hyperparameter Sweep

A systematic sweep was performed by varying the beta parameter (0.01, 0.1 … 0.9) for each loss type while keeping all other settings constant. The sweep was launched on the Hugging Face GPU cluster using a Bash script that iterates over model configurations, loss types, and β values, submitting each run as a separate SLURM job.

configs=("zephyr" "openhermes")
loss_types=("sigmoid" "kto_pair" "ipo")
betas=("0.01" "0.1" "0.2" "0.3" "0.4" "0.5" "0.6" "0.7" "0.8" "0.9")
# ... loop omitted for brevity ...

Results

Zephyr‑7b‑beta‑SFT

  • The lowest β (0.01) yielded the highest MT‑Bench scores for all three algorithms.
  • DPO achieved the best overall MT‑Bench score, but KTO (paired) matched or exceeded DPO in every setting except one.
  • IPO performed worse than the base Zephyr model in most configurations, despite its theoretical robustness.
  • Category‑wise analysis shows notable gaps in Reasoning, Coding, and Math.

OpenHermes‑2.5‑Mistral‑7B

  • The ranking of algorithms remained DPO > KTO > IPO.
  • Optimal β values differed dramatically:
    • DPO: β = 0.6
    • KTO: β = 0.3
    • IPO: β = 0.01
  • Preference alignment improved MT‑Bench by only ~0.3 points, indicating that OpenHermes is already a strong base model.

Both model families demonstrate that β is a critical hyper‑parameter; small changes can swing performance by several MT‑Bench points.

Summary & Insights

  • Careful β tuning is essential for all three alignment methods.
  • In paired‑preference settings, DPO consistently outperforms KTO and IPO, though KTO can be competitive when β is well‑chosen.
  • IPO, while offering convergence guarantees, did not surpass the baseline in these experiments.
  • The open‑source code, configurations, and resulting model checkpoints are publicly available in the Hugging Face alignment‑handbook and the associated model collection.

What’s Next?

Future work will continue adding new preference‑alignment algorithms to 🤗 TRL and expanding the evaluation suite. The authors anticipate that DPO will remain the most robust choice for now, while KTO offers a promising path for leveraging cheap binary feedback without paired data.

Links

Sources