Principle:OpenGVLab InternVL Mixed Preference Optimization

Knowledge Sources	DPO InternVL 2.5 InternVL
Domains	Alignment, Reinforcement_Learning, Vision_Language
Last Updated	2026-02-07 00:00 GMT

Overview

A composite preference optimization technique that combines multiple loss functions (sigmoid DPO, BCO pair, and NLL) to align vision-language models with human preferences.

Description

Mixed Preference Optimization (MPO) extends Direct Preference Optimization (DPO) for multimodal models by combining multiple loss objectives into a single training signal. Standard DPO trains models to prefer chosen responses over rejected ones using a sigmoid loss. MPO augments this with:

• Sigmoid DPO loss: Standard preference optimization based on implicit reward differences
• BCO pair loss: Binary classifier optimization that provides an additional preference signal
• NLL loss (via rpo_alpha): Standard next-token prediction loss on chosen responses, ensuring the model maintains generation quality

The composite loss is: ${ℒ}=w_{sigmoid}\cdot {{ℒ}}_{DPO}+w_{bco}\cdot {{ℒ}}_{BCO}+{\alpha }_{rpo}\cdot {{ℒ}}_{NLL}$

MPO also requires a frozen reference model — an identical copy of the policy model that provides the baseline log-probabilities for the KL divergence term in the DPO objective.

Usage

Use MPO after supervised fine-tuning to align the model with human preferences for response quality, correctness, and safety. It requires preference data with chosen/rejected response pairs.

Theoretical Basis

The DPO objective avoids explicit reward modeling by reparameterizing the reward function:

${{ℒ}}_{DPO}(\theta )=-{\mathbb{𝔼}}_{(x,y_w,y_l)}[\log \sigma (\beta \log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{ref}(y_w|x)}-\beta \log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{ref}(y_l|x)})]$

Where:

• ${\pi }_{\theta }$ is the policy model (trainable)
• ${\pi }_{ref}$ is the reference model (frozen)
• $y_w,y_l$ are chosen and rejected responses
• $\beta$ is the temperature parameter

The BCO pair loss adds a binary classification term, and the NLL loss ensures the model does not degrade in generation quality while optimizing for preferences.

InternVL's default MPO configuration:

• $w_{sigmoid}=0.8$
• $w_{bco}=0.2$
• ${\alpha }_{rpo}=1.0$

Related Pages

Implemented By

• Implementation:OpenGVLab_InternVL_MultimodalDPOTrainer