Principle:Pyro ppl Pyro Evidence Lower Bound
| Metadata | |
|---|---|
| Sources | Automated Variational Inference in Probabilistic Programming, Black Box Variational Inference |
| Domains | Variational_Inference, Optimization |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
The Evidence Lower Bound (ELBO) is the fundamental optimization objective for variational inference in Pyro, providing a tractable lower bound on the log marginal likelihood of observed data.
Description
The ELBO provides a principled way to perform approximate Bayesian inference when the true posterior distribution p(z|x) is intractable. Given a model with observed variables x and latent variables z, and an approximate posterior (guide) q(z), the ELBO is defined as:
This expression can be equivalently written as:
Since the KL divergence is always non-negative, the ELBO is a lower bound on the log evidence log p(x). Maximizing the ELBO with respect to the variational parameters of q(z) is therefore equivalent to minimizing the KL divergence between the approximate posterior and the true posterior.
In Pyro, the ELBO is estimated via Monte Carlo sampling of execution traces. The model and guide are executed to produce a pair of traces, and the ELBO is computed as the difference between the total log probability of the model trace and the total log probability of the guide trace. The Trace_ELBO class implements this single-sample Monte Carlo estimate, optionally using multiple particles to reduce variance. Partial Rao-Blackwellization is applied automatically when non-reparameterizable random variables are present, using conditional independence information from pyro.plate contexts.
Usage
Use the ELBO as the loss function for stochastic variational inference (SVI) in Pyro. The ELBO is the default and most common objective for training probabilistic models where exact posterior inference is intractable. Choose the appropriate ELBO variant based on your model structure: Trace_ELBO for continuous latent variables, TraceGraph_ELBO for finer-grained variance reduction, and TraceEnum_ELBO when discrete latent variables can be enumerated.
Theoretical Basis
The ELBO derivation proceeds from Jensen's inequality applied to the log marginal likelihood:
The gap between the ELBO and the log evidence is exactly the KL divergence:
The Monte Carlo gradient estimator for the ELBO uses the reparameterization trick for reparameterizable distributions (reducing variance) and the score function estimator (REINFORCE) with Rao-Blackwellization for non-reparameterizable distributions.