Principle:LaurentMazare Tch rs Variational Autoencoder

Knowledge Sources	LaurentMazare_Tch_rs Kingma & Welling, 2013 Rezende et al., 2014
Domains	Deep Learning, Generative Models, Probabilistic Inference
Last Updated	2026-02-08 00:00 GMT

Overview

Variational autoencoders learn smooth latent representations by encoding inputs into distributional parameters and training with a combined reconstruction and regularization objective.

Description

A variational autoencoder (VAE) is a generative model that learns to map data into a structured latent space. Unlike a standard autoencoder that encodes inputs to fixed latent vectors, a VAE encodes each input to the parameters of a probability distribution -- specifically, a mean vector and a variance (or log-variance) vector that define a Gaussian distribution in latent space.

The reparameterization trick is the key technical innovation that makes VAEs trainable via backpropagation. Instead of sampling directly from the encoded distribution (which would block gradient flow), the model samples from a standard normal distribution and then shifts and scales the sample using the learned mean and variance. This reformulates the stochastic sampling as a deterministic function of the parameters plus independent noise: $z=\mu +\sigma \odot ϵ$, where $ϵ\sim {𝒩}(0,I)$.

The decoder reconstructs the input from the sampled latent vector. The total loss combines a reconstruction term (how well the decoder output matches the input) with a KL divergence term (how far the encoded distribution deviates from a standard normal prior). This KL term acts as a regularizer, ensuring the latent space is smooth and continuous.

Usage

Apply the VAE principle when:

• Generating new samples that resemble training data (images, text, molecules)
• Learning disentangled or interpretable latent representations
• Performing smooth interpolation between data points in latent space
• Building probabilistic generative models with tractable inference

Theoretical Basis

Evidence Lower Bound (ELBO)

The VAE maximizes the evidence lower bound on the log-likelihood of the data:

${ℒ}(\theta ,\varphi ;x)={\mathbb{𝔼}}_{q_{\varphi }(z|x)}[\log p_{\theta }(x|z)]-D_{KL}(q_{\varphi }(z|x)\Vert p(z))$

where:

• $q_{\varphi }(z|x)$ is the encoder (approximate posterior)
• $p_{\theta }(x|z)$ is the decoder (likelihood)
• $p(z)={𝒩}(0,I)$ is the prior

Reparameterization Trick

To enable gradient-based optimization through the sampling step:

$z={\mu }_{\varphi }(x)+{\sigma }_{\varphi }(x)\odot ϵ,ϵ\sim {𝒩}(0,I)$

This makes $z$ a differentiable function of $\varphi$, allowing standard backpropagation.

KL Divergence (Closed Form)

When both the approximate posterior and prior are Gaussian, the KL divergence has a closed-form solution:

$D_{KL}(q\Vert p)=-\frac{1}{2}\sum _{j=1}^J(1+\log ({\sigma }_j^2)-{\mu }_j^2-{\sigma }_j^2)$

where $J$ is the dimensionality of the latent space.

Reconstruction Loss

For binary or normalized data, the reconstruction term is typically the binary cross-entropy between the input and the decoder output. For continuous data, mean squared error is common.

Related Pages

• Implementation:LaurentMazare_Tch_rs_VAE_Example