Principle:Danijar Dreamerv3 Agent Initialization

Knowledge Sources	Mastering Diverse Domains through World Models Dream to Control: Learning Behaviors by Latent Imagination DreamerV3
Domains	Reinforcement_Learning, Model_Based_RL, World_Models
Last Updated	2026-02-15 09:00 GMT

Overview

Construction of a world-model-based RL agent that wires together a Recurrent State-Space Model (RSSM), encoder/decoder pair, reward and continue predictors, actor-critic policy, and optimizer into a unified trainable system.

Description

Agent Initialization in DreamerV3 assembles the complete model-based RL architecture. The agent comprises:

• RSSM World Model (self.dyn): Learns latent dynamics with discrete stochastic states (32x32 categorical) plus a deterministic GRU-like recurrent core (4096 units)
• Encoder (self.enc): Maps observations (images via CNN, vectors via MLP) to tokens
• Decoder (self.dec): Reconstructs observations from latent features
• Reward Head (self.rew): Predicts rewards from latent features
• Continue Head (self.con): Predicts episode continuation probability
• Policy Head (self.pol): Outputs action distributions from latent features
• Value Head (self.val): Estimates expected returns; slow target copy (self.slowval) for stable training
• Normalizers (retnorm, valnorm, advnorm): Percentile-based running statistics for scale-free learning
• Optimizer (self.opt): Custom optimizer with adaptive gradient clipping (AGC), RMS scaling, and momentum

This design enables the agent to learn entirely in imagination — after learning world dynamics from data, the policy is optimized through imagined rollouts without additional environment interaction.

Usage

Use this principle as the third step in any DreamerV3 workflow, after configuration loading and environment construction. The agent must be initialized before checkpoint restoration, training, or evaluation can begin.

Theoretical Basis

The DreamerV3 agent architecture follows the Dreamer family design:

$\text{World Model: }{s}_t=f_{\theta }(s_{t-1},a_{t-1},o_t)$

$\text{Imagination: }{\hat{s}}_{t+k}=f_{\theta }({\hat{s}}_{t+k-1},\pi ({\hat{s}}_{t+k-1}))$

$\text{Policy Optimization: }\underset{\pi }{\max }{\mathbb{𝔼}}_{f_{\theta },\pi }[\sum _{k=0}^H{\gamma }^{k}r({\hat{s}}_{t+k})]$

Pseudo-code Logic:

# Abstract algorithm
encoder = CNN_MLP_Encoder(obs_space)
world_model = RSSM(act_space, deter=4096, stoch=32x32)
decoder = CNN_MLP_Decoder(obs_space)
reward_head = MLP(scalar)
continue_head = MLP(binary)
policy = MLP(act_space)
value = MLP(scalar)
slow_value = EMA_copy(value)
optimizer = AGC_RMS_Momentum(all_modules)

Related Pages

Implemented By

• Implementation:Danijar_Dreamerv3_Agent_Init