Principle:Danijar Dreamerv3 Distributed Replay Management
| Knowledge Sources | |
|---|---|
| Domains | Reinforcement_Learning, Distributed_Systems, Experience_Replay |
| Last Updated | 2026-02-15 09:00 GMT |
Overview
A dedicated replay process that manages experience storage and sampling as an RPC service, with rate limiting to enforce a target samples-per-insert ratio in distributed training.
Description
In distributed DreamerV3 training, the replay buffer runs as an independent process serving RPC requests from actors (inserting transitions) and learners (sampling batches). A SamplesPerInsert rate limiter enforces the configured train ratio by blocking inserts when sampling falls behind or blocking samples when inserts fall behind.
The replay process manages:
- Two replay buffers: Training and evaluation
- Three data streams: train, report, eval (each providing batched sequences)
- Rate limiting: SamplesPerInsert with configurable tolerance
- RPC server: Exposes add_batch, sample_batch_train, sample_batch_report, sample_batch_eval, and update methods
- Checkpointing: Periodically saves replay state to disk
Usage
This principle applies only in distributed (parallel) training mode. The replay process is spawned by the combined launcher and runs independently, coordinating with actor and learner processes via portal RPC.
Theoretical Basis
Rate Limiting Logic:
# Abstract algorithm
limiter = SamplesPerInsert(
ratio=train_ratio / batch_length,
tolerance=4 * batch_size)
# On insert:
wait_until(limiter.want_insert) # Block if too many inserts ahead
limiter.insert()
replay.add(transition)
# On sample:
wait_until(limiter.want_sample) # Block if too many samples ahead
limiter.sample()
batch = stream.next()
The limiter ensures that the ratio of sample calls to insert calls stays near the target train_ratio / batch_length, preventing either the actor or learner from running too far ahead.