Principle:OpenRLHF OpenRLHF PPO Value Loss

Knowledge Sources	Proximal Policy Optimization Algorithms
Domains	Reinforcement_Learning, Loss_Functions
Last Updated	2026-02-07 00:00 GMT

Overview

A clipped value function loss that trains the critic model to predict expected returns while constraining updates relative to previous value estimates.

Description

PPO Value Loss trains the critic (value function) model by minimizing the squared error between predicted values and computed returns, optionally with value clipping to prevent large value function changes between updates. This ensures stable advantage estimation for policy optimization.

Usage

Used as the critic loss in PPO training. Paired with PolicyLoss for the actor.

Theoretical Basis

Clipped value loss: $L^V=\frac{1}{2}\max [(V_{\theta }-R{)}^2,(\text{clip}(V_{\theta },V_{old}\pm ϵ)-R{)}^2]$

where $R$ is the return (computed via GAE) and $ϵ$ is the clip range.

Related Pages

Implemented By

• Implementation:OpenRLHF_OpenRLHF_ValueLoss