Principle:LaurentMazare Tch rs Deep Deterministic Policy Gradient

Knowledge Sources	LaurentMazare_Tch_rs control with deep reinforcement learning Lillicrap et al., 2016 Policy Gradient Algorithms Silver et al., 2014
Domains	Reinforcement Learning, Continuous Control, Actor-Critic Methods
Last Updated	2026-02-08 00:00 GMT

Overview

Deep Deterministic Policy Gradient (DDPG) is an off-policy, model-free reinforcement learning algorithm that extends Q-learning to continuous action spaces by learning a deterministic policy alongside an action-value function.

Description

Classical reinforcement learning algorithms like DQN work well for discrete action spaces (e.g., choosing from a finite set of buttons), but many real-world control problems have continuous action spaces (e.g., applying a torque value anywhere in a range). DDPG addresses this gap by combining ideas from several areas:

• Actor-Critic architecture - Two neural networks work in tandem: the actor (policy network) $\mu (s|{\theta }^{\mu })$ maps states directly to continuous actions, and the critic (value network) $Q(s,a|{\theta }^Q)$ estimates the expected return of taking action $a$ in state $s$.

• Deterministic policy - Unlike stochastic policy gradient methods that output action probability distributions, DDPG's actor outputs a single deterministic action for each state. This is more sample-efficient for continuous spaces because it avoids integrating over the action space.

• Off-policy learning with experience replay - Transitions $(s,a,r,s^{\prime })$ are stored in a replay buffer and sampled randomly for training. This breaks temporal correlations, improves sample efficiency, and allows learning from past experience.

• Target networks - Separate target copies of both actor and critic networks are maintained for computing TD targets. These target networks are updated slowly via soft updates (Polyak averaging), which stabilizes training by preventing the learning targets from changing too rapidly.

• Ornstein-Uhlenbeck exploration noise - Since the deterministic policy produces no exploration on its own, noise is added to actions during training. The Ornstein-Uhlenbeck (OU) process generates temporally correlated noise, which is physically plausible for inertial systems and provides smoother exploration than independent Gaussian noise.

Usage

DDPG is appropriate for:

• Continuous control tasks - Robotic manipulation, locomotion, autonomous driving, where actions are real-valued vectors.
• Low-dimensional action spaces - DDPG works best when the action space has moderate dimensionality (roughly 1-20 dimensions).
• Environments with smooth dynamics - The deterministic policy and OU noise work best when the optimal policy is smooth with respect to the state.

DDPG may not be suitable for:

• Discrete action spaces - DQN or policy gradient methods are more natural.
• Very high-dimensional action spaces - Scalability becomes an issue.
• Highly stochastic environments - A stochastic policy may be fundamentally necessary.

Modern successors like TD3 and SAC address several of DDPG's stability issues while retaining its core ideas.

Theoretical Basis

Deterministic Policy Gradient Theorem

The key theoretical result (Silver et al., 2014) shows that the gradient of the expected return $J$ with respect to the policy parameters ${\theta }^{\mu }$ is:

${\mathrm{\nabla }}_{{\theta }^{\mu }}J={\mathbb{𝔼}}_{s\sim {\rho }^{\mu }}[{\mathrm{\nabla }}_{{\theta }^{\mu }}\mu (s|{\theta }^{\mu })\cdot {\mathrm{\nabla }}_{a}Q(s,a|{\theta }^Q){|}_{a=\mu (s)}]$

where ${\rho }^{\mu }$ is the state distribution under policy $\mu$. This avoids integrating over the action space, requiring only the gradient of $Q$ with respect to the action, evaluated at the action chosen by the current policy.

Critic Update

The critic is trained to minimize the Bellman error using transitions $(s_i,a_i,r_i,s{\text{'}}_i)$ sampled from the replay buffer:

$y_i=r_i+\gamma Q^{\prime }(s{\text{'}}_i,{\mu }^{\prime }(s{\text{'}}_i|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }})$

$L=\frac{1}{N}\sum _i(y_i-Q(s_i,a_i|{\theta }^Q){)}^2$

where $Q^{\prime }$ and ${\mu }^{\prime }$ are the target networks and $\gamma$ is the discount factor.

Actor Update

The actor is updated by ascending the deterministic policy gradient, approximated using a minibatch:

${\mathrm{\nabla }}_{{\theta }^{\mu }}J\approx \frac{1}{N}\sum _i{\mathrm{\nabla }}_{{\theta }^{\mu }}\mu (s_i|{\theta }^{\mu })\cdot {\mathrm{\nabla }}_{a}Q(s_i,a|{\theta }^Q){|}_{a=\mu (s_i)}$

In practice, this is computed by:

1. Forward pass the state through the actor to get action $a=\mu (s)$.
2. Forward pass $(s,a)$ through the critic to get $Q(s,a)$.
3. Backpropagate through both networks, but only update the actor's parameters.

Soft Target Updates

Target networks are updated using Polyak averaging with coefficient $\tau \ll 1$ (typically $\tau =0.001$):

${\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}$ ${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}$

This creates a slowly-moving target that stabilizes the TD learning process, analogous to the fixed target network in DQN but smoother.

Ornstein-Uhlenbeck Noise Process

The OU process is defined by the stochastic differential equation:

$d{x}_t={\theta }_{OU}({\mu }_{OU}-x_t)dt+{\sigma }_{OU}d{W}_t$

where:

• ${\theta }_{OU}$ is the mean reversion rate (how fast noise returns to the mean)
• ${\mu }_{OU}$ is the long-term mean (typically 0)
• ${\sigma }_{OU}$ is the volatility (noise magnitude)
• $W_t$ is a Wiener process (Brownian motion)

In discrete time with step size $\mathrm{\Delta }t=1$:

x_{t+1} = x_t + theta * (mu - x_t) + sigma * N(0, 1)

The exploration action is then:

$a_t=\mu (s_t|{\theta }^{\mu })+x_t$

Experience Replay Buffer

The replay buffer stores the most recent $M$ transitions as a circular buffer:

BUFFER of capacity M
FUNCTION store(s, a, r, s', done):
    buffer[position] = (s, a, r, s', done)
    position = (position + 1) mod M

FUNCTION sample(batch_size):
    indices = random_integers(0, current_size, batch_size)
    RETURN buffer[indices]

Complete DDPG Training Loop

INITIALIZE actor mu, critic Q with random weights
INITIALIZE target networks mu', Q' as copies
INITIALIZE replay buffer B
INITIALIZE OU noise process

FOR each episode:
    RESET environment, get initial state s
    RESET noise process

    FOR each step:
        a = mu(s) + noise.sample()
        s', r, done = environment.step(a)
        B.store(s, a, r, s', done)

        IF B.size >= batch_size:
            Sample minibatch from B
            Compute target y = r + gamma * Q'(s', mu'(s'))
            Update critic by minimizing (y - Q(s,a))^2
            Update actor using policy gradient
            Soft-update target networks

        s = s'
        IF done: BREAK

Related Pages

• Implementation:LaurentMazare_Tch_rs_DDPG_Agent