TRL adds Direct Preference Optimization support for Vision‑Language Models

TL;DR

Hugging Face announced that the TRL library now supports Direct Preference Optimization (DPO) for Vision‑Language Models (VLMs), enabling developers to fine‑tune models such as Idefics‑2, Llava 1.5, and PaliGemma with preference data while keeping memory requirements manageable through bfloat16 quantization and LoRA adapters.

Preference‑based fine‑tuning for VLMs

Preference optimization replaces costly label‑wise supervision with binary comparisons: each training example contains a prompt, a chosen answer, and a rejected answer. The model learns to assign higher probability to the chosen response. This approach captures nuanced human judgments and has been widely adopted for language models; the new TRL integration extends it to multimodal VLMs.

Example dataset

The blog uses the openbmb/RLAIF‑V‑Dataset, which provides 83 k+ rows of image‑question pairs with chosen and rejected textual answers. A sample entry looks like:

Question: "How many families?"
Rejected: "The image does not provide any information about families."
Chosen:   "The image shows a Union Organization table setup with 18,000 families."

The chosen answer may still be factually incorrect, but it is less wrong than the rejected answer, which is the core premise of preference learning.

Formatting for chat‑style VLMs

The dataset must be reshaped into a chat format where the user supplies an image and a textual query, and the assistant replies with either the chosen or rejected text. The Hugging Face AutoProcessor (e.g., HuggingFaceM4/idefics2-8b) is used to apply the chat template and to resize images to the processor’s maximum edge length, preventing out‑of‑memory (OOM) errors. The code snippet below illustrates the transformation:

from datasets import features
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b", do_image_splitting=False)

def format(example):
    prompt = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": example["question"]}]}]
    chosen = [{"role": "assistant", "content": [{"type": "text", "text": example["chosen"]}]}]
    rejected = [{"role": "assistant", "content": [{"type": "text", "text": example["rejected"]}]}]
    prompt   = processor.apply_chat_template(prompt,   tokenize=False)
    chosen   = processor.apply_chat_template(chosen,   tokenize=False)
    rejected = processor.apply_chat_template(rejected, tokenize=False)
    max_size = processor.image_processor.size["longest_edge"]
    example["image"].thumbnail((max_size, max_size))
    return {"images": [example["image"]], "prompt": prompt, "chosen": chosen, "rejected": rejected}

After mapping this function over the dataset and casting the images column to decoded PIL.Image objects, the data is ready for training.

Training a VLM with DPO

The blog demonstrates fine‑tuning Idefics‑2‑8b as a reference model, but the same pipeline works for Llava 1.5 and PaliGemma.

Memory budgeting

Training a full‑precision 8 B‑parameter model requires roughly 160 GB of VRAM (model, reference copy, gradients, and AdamW states). The authors show a step‑by‑step calculation:

Component Bytes per param Total (GB)
Model (train) 4 (float32) 32
Reference model 4 32
Gradients 4 32
Optimizer states (2×) 4 64
Total – 160

Because most GPUs are far smaller, the blog recommends two complementary techniques.

Quantization to bfloat16

Switching to torch.bfloat16 halves the per‑parameter storage from 4 bytes to 2 bytes, cutting model memory from 32 GB to 16 GB. The change is applied both to the model and the optimizer:

model = AutoModelForVision2Seq.from_pretrained(..., torch_dtype=torch.bfloat16)
training_args = DPOConfig(..., bf16=True)

LoRA adapters via PEFT

Low‑Rank Adaptation (LoRA) freezes the base model and injects trainable rank‑decomposition matrices into linear layers. Using peft.LoraConfig(target_modules="all-linear") reduces trainable parameters from 8 B to ~55 M (≈0.65 % of the total). Memory for gradients and optimizer states now drops to a few hundred megabytes.

Re‑computing the budget after quantization and LoRA yields ≈32 GB total, comfortably fitting an 80 GB GPU.

Batch size and activation memory

Activations are not accounted for in the static budget. The authors suggest an empirical approach: start with a desired batch size (e.g., 64), observe OOM, halve the batch size, and double gradient_accumulation_steps to keep the effective batch size constant. In their experiment they settled on per_device_train_batch_size=2 with gradient_accumulation_steps=32. Enabling gradient_checkpointing=True further reduces activation memory at the cost of extra compute.

Complete training script

A self‑contained script (dpo_idefics2-8b.py) ties together model loading, dataset formatting, LoRA configuration, and the DPOTrainer. Key arguments include:

  • bf16=True and gradient_checkpointing=True
  • per_device_train_batch_size=2, gradient_accumulation_steps=32
  • Parallel preprocessing with dataset_num_proc=32 and dataloader_num_workers=32
  • LoraConfig(target_modules="all-linear") passed to DPOTrainer

Running the script with accelerate launch dpo_idefics2-8b.py launches a single‑epoch DPO fine‑tune.

Training outcomes

The loss curves show steady improvement in two DPO‑specific metrics:

  • Accuracy – proportion of samples where the model assigns higher probability to the chosen answer.
  • Reward margin – difference between the reward (log‑probability) of the chosen and rejected answers; a growing margin indicates successful preference learning.

Both metrics increase over the training horizon, confirming that DPO can effectively steer VLMs toward preferred responses.

Evaluation on hallucination reduction

To assess whether DPO mitigates hallucinations, the fine‑tuned Idefics‑2 model was evaluated on the AMBER benchmark (a VLM‑specific hallucination test). Results (accuracy / F1) are:

Model Accuracy F1
GPT‑4o 88.8 91.6
Idefics‑2 + DPO 85.9 89.4
Idefics‑2 (baseline) 85.8 89.1
GPT‑4v 83.4 87.4
MiniGemini 82.6 87.6
… … …

The DPO‑fine‑tuned model matches or slightly exceeds the baseline, indicating a modest reduction in hallucinations.

Qualitative examples

Selected AMBER samples illustrate the change:

Image Question Baseline Idefics‑2 Idefics‑2 + DPO
![ships] Are there two ships? Yes No
![ground] Is the ground uneven? No Yes
![shovel] Is there one shovel? Yes No

These examples show the model learning to prefer the less‑hallucinatory answer when the training data signals a preference.

Extending DPO to other VLMs

TRL’s DPO implementation already supports Llava 1.5 and PaliGemma. The blog points to an example script in the TRL repo (examples/scripts/dpo_vlm.py). For PaliGemma, a typical command line looks like:

accelerate launch examples/scripts/dpo_visual.py \
    --dataset_name HuggingFaceH4/rlaif-v_formatted \
    --model_name_or_path google/paligemma-3b-pt-224 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 32 \
    --dataset_num_proc 32 \
    --output_dir dpo_paligemma_rlaif-v \
    --bf16 \
    --torch_dtype bfloat16 \
    --gradient_checkpointing \
    --use_peft \
    --lora_target_modules=all-linear

The same quantization‑+‑LoRA recipe applies, making DPO accessible on modest GPU hardware.

Implications

By integrating DPO into TRL for VLMs, Hugging Face lowers the barrier to preference‑driven fine‑tuning of multimodal models. Developers can now align VLMs with human judgments without expensive label collection, while staying within the memory limits of a single high‑end GPU. The modest hallucination improvements on AMBER suggest that preference data can be an effective signal for reducing over‑confident errors, opening a path toward safer, more trustworthy vision‑language assistants.

TL;DR – The new TRL DPO support lets you fine‑tune vision‑language models using binary preference data, and with bfloat16 quantization plus LoRA adapters you can train an 8 B‑parameter VLM on a single 80 GB GPU, achieving measurable gains in preference accuracy and reduced hallucinations.

Sources