Principle:LaurentMazare Tch rs Advantage Actor Critic

Knowledge Sources	LaurentMazare_Tch_rs Asynchronous Methods for Deep Reinforcement Learning
Domains	Reinforcement Learning, Deep Learning
Last Updated	2026-02-08 00:00 GMT

Overview

Advantage Actor-Critic (A2C) is a synchronous reinforcement learning algorithm that combines a policy network (actor) with a value network (critic) to reduce variance in policy gradient estimation through advantage-based updates.

Description

A2C is the synchronous variant of the Asynchronous Advantage Actor-Critic (A3C) algorithm. It uses two coordinated components:

• Actor (policy network): Outputs a probability distribution over actions given the current state, $\pi (a|s;\theta )$. The actor is trained to select actions that maximize expected cumulative reward.

• Critic (value network): Estimates the state value function $V(s;\varphi )$, which represents the expected cumulative reward from state $s$ onward under the current policy. The critic provides a baseline that reduces the variance of policy gradient estimates without introducing bias.

• Advantage function: The advantage $A(s,a)=R-V(s)$ measures how much better an action is compared to the average action in that state. Using advantage rather than raw returns dramatically reduces the variance of gradient estimates, since the critic absorbs the state-dependent portion of the return.

• Entropy bonus: An entropy regularization term is added to the policy loss to encourage exploration. This prevents the policy from collapsing prematurely to a deterministic action selection, which could cause the agent to get stuck in local optima.

The synchronous nature of A2C means that multiple environment instances are stepped in lockstep, and their experiences are batched together for a single gradient update. This is simpler than asynchronous variants and often achieves comparable performance with better GPU utilization.

Usage

A2C is applied to discrete and continuous control tasks in reinforcement learning, including game playing, robotic control, and navigation. It serves as a foundational algorithm from which more advanced methods (PPO, IMPALA) are derived.

Theoretical Basis

Policy Gradient Theorem:

${\mathrm{\nabla }}_{\theta }J(\theta )={\mathbb{𝔼}}_{{\pi }_{\theta }}[{\mathrm{\nabla }}_{\theta }\log \pi (a_t|s_t;\theta )\cdot A(s_t,a_t)]$

Advantage Estimation:

Using n-step returns:

$A(s_t,a_t)=\left(\sum _{k=0}^{n-1}{\gamma }^k{r}_{t+k}\right)+{\gamma }^{n}V(s_{t+n};\varphi )-V(s_t;\varphi )$

where $\gamma$ is the discount factor. For a complete episode, the bootstrapped value term ${\gamma }^{n}V(s_{t+n})$ is replaced with the actual terminal return.

Actor Loss (Policy Loss):

${{ℒ}}_{actor}=-{\mathbb{𝔼}}_t[\log \pi (a_t|s_t;\theta )\cdot A_t]$

Critic Loss (Value Loss):

${{ℒ}}_{critic}={\mathbb{𝔼}}_t[(R_t-V(s_t;\varphi ){)}^2]$

where $R_t$ is the n-step return target.

Entropy Bonus:

$H(\pi (\cdot |s_t))=-\sum _a\pi (a|s_t)\log \pi (a|s_t)$

Combined Loss:

${ℒ}={{ℒ}}_{actor}+c_v\cdot {{ℒ}}_{critic}-c_e\cdot H(\pi )$

where $c_v$ is the value loss coefficient and $c_e$ is the entropy coefficient.

Synchronous Batching:

With $N$ parallel environments, each contributing $T$ steps:

for each update iteration:
    collect T steps from N parallel environments
    compute advantages for all N*T transitions
    compute combined loss over the batch
    perform single gradient update

Related Pages

• Implementation:LaurentMazare_Tch_rs_A2C_Agent