Principle:Volcengine Verl Parallel Actor Critic Update
| Knowledge Sources | |
|---|---|
| Domains | Distributed_Training, Optimization, Reinforcement_Learning |
| Last Updated | 2026-02-07 18:00 GMT |
Overview
A training loop optimization that overlaps actor and critic gradient update steps by issuing non-blocking RPC calls to separate worker groups.
Description
Parallel Actor-Critic Update is a training loop modification that exploits split resource placement to overlap the two heaviest compute phases in PPO training: the actor policy update and the critic value function update. In the standard sequential loop, these run one after the other. When actor and critic reside on separate GPU pools, both updates can be issued as non-blocking remote calls and then awaited together.
This principle modifies the inner training loop:
- Standard: update_actor() → wait → update_critic() → wait
- Parallel: update_actor(non-blocking) → update_critic(non-blocking) → wait for both
The wall-clock time for the update phase drops from (actor_time + critic_time) to max(actor_time, critic_time).
Usage
Use this principle in conjunction with Split Resource Placement when you want to minimize training step latency. It requires that actor and critic worker groups are on separate resource pools so their GPU operations do not contend.
Theoretical Basis
Pseudo-code Logic:
# Abstract parallel update (NOT real implementation)
# Sequential (standard):
actor_output = actor_wg.update_actor(batch) # blocks
critic_output = critic_wg.update_critic(batch) # blocks
# Total time: T_actor + T_critic
# Parallel (split placement):
actor_future = actor_wg.update_actor(batch) # non-blocking
critic_future = critic_wg.update_critic(batch) # non-blocking
actor_output = actor_future.get() # wait
critic_output = critic_future.get() # wait
# Total time: max(T_actor, T_critic)