Vision Language Model Alignment in TRL

Hugging Face has introduced comprehensive support for advanced alignment techniques for Vision Language Models (VLMs) within the TRL library. This update enables developers to move beyond simple pairwise preference optimization to more scalable and robust methods like Mixed Preference Optimization (MPO), Group Relative Policy Optimization (GRPO), and Group Sequence Policy Optimization (GSPO).

Advanced Multimodal Alignment Methods

TRL now integrates several cutting-edge alignment algorithms designed to extract richer signals from preference data and improve VLM performance on complex tasks.

Mixed Preference Optimization (MPO)

MPO addresses the distribution shift common in Supervised Fine-Tuning (SFT) for reasoning tasks and the lack of coherent rationales often found in Direct Preference Optimization (DPO). MPO extends DPO by combining three distinct losses:

  • Preference loss from DPO (sigmoid)
  • Quality loss from Binary Classifier Optimization (BCO)
  • Generation loss from SFT

According to the source paper, implementing this combined loss can lead to a 6.2 point improvement on the MathVista benchmark. In TRL, this is implemented via the DPOTrainer by configuring loss_type with ["sigmoid", "bco_pair", "sft"] and assigning corresponding weights.

Multimodal Group Relative Policy Optimization (GRPO)

GRPO is a reinforcement learning alignment method that performs policy updates over groups (batches of trajectories) rather than individual samples. This approach makes the model more robust to reward noise, as noise averages out within the groups, allowing the model to learn a broader sense of what constitutes a high-quality response.

To implement GRPO in TRL, users define reward functions—such as one for format validation (e.g., checking for <think> and <answer> tags) and one for accuracy (verifying the solution against ground truth)—and pass them to the GRPOTrainer.

Group Sequence Policy Optimization (GSPO)

GSPO is a variant of GRPO developed by Qwen to improve training stability. It achieves this by computing importance sampling weights at the sequence level instead of the token level, a benefit particularly relevant for Mixture-of-Experts (MoE) style models. TRL supports GSPO through the GRPOConfig, allowing users to specify importance_sampling_level="sequence" along with specific epsilon and beta parameters.

Extended VLM Support and SFT

Beyond the primary new algorithms, TRL has expanded its support for existing alignment and fine-tuning workflows.

RLOO and Online DPO

TRL now supports Reinforce Leave One Out (RLOO) and Online Direct Preference Optimization (Online DPO) for VLMs. These methods enable alignment on multimodal datasets and are accessible via the RLOOTrainer and OnlineDPOTrainer respectively.

Native Supervised Fine-tuning (SFT)

With the standardization of the transformers API, SFTTrainer now provides full native support for VLMs. Users can initialize the trainer with a VLM and a dataset containing an images column. To prevent the removal of image tokens during training, it is recommended to set max_length=None in the SFTConfig.

vLLM Integration for Online Alignment

To support online alignment methods that require generating samples during the training loop, TRL integrates vLLM. This integration offers two primary operational modes:

  • Colocate mode: Runs vLLM in the same process as the training loop, sharing the GPU between training and generation.
  • Server mode: Requires vLLM to be served as a separate process, which the training script then queries.

Additionally, TRL now supports using the transformers backend with vLLM, which can be enabled via the --vllm_model_impl transformers flag.

Sources