Principle:Farama Foundation Gymnasium Generalized Advantage Estimation

Knowledge Sources	GAE Sutton and Barto RL
Domains	Reinforcement_Learning, Policy_Gradient
Last Updated	2026-02-15 03:00 GMT

Overview

A variance reduction technique for policy gradient methods that computes advantage estimates using an exponentially weighted sum of temporal-difference residuals.

Description

Generalized Advantage Estimation (GAE) addresses the bias-variance tradeoff in advantage estimation for policy gradient algorithms. Raw Monte Carlo returns have high variance, while single-step TD errors have high bias. GAE interpolates between these extremes using a parameter $\lambda \in [0,1]$:

• $\lambda =0$: Single-step TD (low variance, high bias)
• $\lambda =1$: Monte Carlo returns (high variance, low bias)

GAE is the standard advantage estimator in PPO, A2C, and other actor-critic methods. It provides smooth control over the bias-variance tradeoff, typically with $\lambda =0.95$ as a widely-used default.

Usage

Use GAE when implementing actor-critic algorithms that require advantage estimates. It is computed after collecting a batch of trajectories from vectorized environments and requires a learned value function for bootstrapping.

Theoretical Basis

The TD residual at time $t$: ${\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t)$

The GAE advantage estimate: ${\hat{A}}_t^{GAE(\gamma ,\lambda )}=\sum _{l=0}^{T-t-1}(\gamma \lambda {)}^l{\delta }_{t+l}$

This can be computed efficiently via backward recursion: ${\hat{A}}_T=0$ ${\hat{A}}_t={\delta }_t+\gamma \lambda {\hat{A}}_{t+1}$

At episode boundaries (terminated=True): $V(s_{t+1})=0$ At truncation (truncated=True): $V(s_{t+1})$ is used for bootstrapping.

Related Pages

Implemented By

• Implementation:Farama_Foundation_Gymnasium_GAE_Computation