Principle:Zai org CogVideo VAE Encode Decode
| Knowledge Sources | |
|---|---|
| Domains | Video_Generation, Variational_Autoencoders, Latent_Space_Modeling |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Variational Autoencoder (VAE) encoding and decoding transforms high-dimensional data (such as video frames) into a compact latent representation and back, enabling efficient generative modeling in a lower-dimensional space.
Description
A Variational Autoencoder consists of two complementary networks: an encoder that maps input data to a probability distribution in latent space, and a decoder that reconstructs data from samples drawn from that distribution. The encoder outputs parameters of a posterior distribution q(z|x), typically a diagonal Gaussian parameterized by a mean vector and a log-variance vector. During encoding, a latent sample is drawn using the reparameterization trick: z = mu + sigma * epsilon, where epsilon is sampled from a standard normal distribution. The decoder then maps this latent sample back to the data space.
For video applications, the VAE operates over spatiotemporal volumes. The encoder compresses frames from pixel space (high resolution, many channels) into a latent tensor with reduced spatial and temporal dimensions. This compressed representation captures the essential visual content while discarding redundant information. The decoder reverses this process, reconstructing full-resolution video frames from the compact latent codes.
To handle the large memory requirements of video data, practical implementations employ two key techniques: slicing (processing the latent space in smaller slices along the batch or channel dimension) and tiling (splitting spatial dimensions into overlapping tiles, encoding/decoding each tile independently, and blending at boundaries).
Usage
Use VAE encode/decode when building latent diffusion models for video generation, when pre-computing latent representations for training efficiency, or when evaluating reconstruction fidelity of a generative model's autoencoder component.
Theoretical Basis
The VAE training objective maximizes the Evidence Lower Bound (ELBO):
ELBO = E_q(z|x)[log p(x|z)] - KL(q(z|x) || p(z))
The first term is the reconstruction loss, encouraging the decoder to faithfully reproduce the input. The second term is the KL divergence between the learned posterior q(z|x) and the prior p(z) (typically a standard normal), which regularizes the latent space to be smooth and continuous.
The reparameterization trick enables backpropagation through the stochastic sampling step by expressing the random variable as a deterministic function of the distribution parameters and an independent noise source:
z = mu + sigma * epsilon, where epsilon ~ N(0, I)
For the KL-regularized variant (AutoencoderKL), the KL divergence has a closed-form solution for Gaussian distributions:
KL = -0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
In practice, the reconstruction loss uses a combination of pixel-wise L1/L2 loss and perceptual loss (comparing features from a pretrained network) to ensure both pixel accuracy and perceptual quality.