Principle:Pyro ppl Pyro Information Form Gaussian
| Knowledge Sources | |
|---|---|
| Domains | Gaussian Inference, Message Passing, Information Theory |
| Last Updated | 2026-02-09 09:00 GMT |
Overview
The information form (canonical form or natural parameter form) of a Gaussian distribution represents the distribution using the precision matrix and precision-weighted mean, enabling efficient multiplication, marginalization, and message passing operations.
Description
A multivariate Gaussian distribution can be represented in two equivalent forms:
Moment form (standard): N(mu, Sigma) parameterized by mean mu and covariance Sigma.
Information form (canonical): parameterized by the precision matrix Lambda = Sigma^{-1} and the information vector eta = Sigma^{-1} * mu.
While the moment form is intuitive for sampling and computing marginals, the information form is superior for several operations central to inference:
Multiplication: When two Gaussian potentials are multiplied (as in Bayes' rule), the information form parameters simply add: Lambda_posterior = Lambda_prior + Lambda_likelihood, eta_posterior = eta_prior + eta_likelihood. In moment form, this requires matrix inversion.
Conditioning: Conditioning on observed variables is a simple slicing operation in information form.
Message passing: In graphical model inference (belief propagation, Kalman filtering), messages between nodes are naturally expressed as Gaussian potentials in information form. The sum-product algorithm becomes simple addition of information parameters.
The GammaGaussian extension couples a Gaussian variable with a Gamma-distributed precision parameter, enabling joint inference over both the mean and the precision of a Gaussian. This is useful in models where the noise variance is unknown.
In Pyro, information form Gaussians are used internally for:
- Exact inference in linear Gaussian models.
- Efficient implementation of Kalman filters and smoothers.
- Message passing in Gaussian graphical models.
- Variable elimination with Gaussian factors.
Usage
Use information form Gaussians when:
- Performing belief propagation or variable elimination in Gaussian graphical models.
- Implementing Kalman filters where prediction and update steps are repeated.
- Multiplying many Gaussian factors together (e.g., combining prior and multiple likelihoods).
- Working with sparse precision matrices (common in spatial and temporal models).
- Need efficient conditioning and marginalization operations.
Theoretical Basis
Information form parameterization:
# Moment form: N(mu, Sigma)
# log p(x) = -0.5 * (x - mu)^T Sigma^{-1} (x - mu) - 0.5 * log|Sigma| - (d/2)*log(2*pi)
# Information form: (eta, Lambda)
# eta = Sigma^{-1} * mu (information vector)
# Lambda = Sigma^{-1} (precision matrix)
# log p(x) = -0.5 * x^T Lambda x + eta^T x - 0.5 * eta^T Lambda^{-1} eta
# + 0.5 * log|Lambda| - (d/2)*log(2*pi)
# Conversion:
# mu = Lambda^{-1} * eta
# Sigma = Lambda^{-1}
Product of Gaussians (Bayes' rule):
# Moment form (expensive):
# Sigma_post = (Sigma_prior^{-1} + Sigma_lik^{-1})^{-1}
# mu_post = Sigma_post * (Sigma_prior^{-1} mu_prior + Sigma_lik^{-1} mu_lik)
# Requires two matrix inversions
# Information form (cheap):
# Lambda_post = Lambda_prior + Lambda_lik
# eta_post = eta_prior + eta_lik
# Just addition!
Marginalization (integrating out a subset of variables):
# Partition x = (x_a, x_b), want to compute p(x_a):
# In information form with block structure:
# Lambda = [[Lambda_aa, Lambda_ab],
# [Lambda_ba, Lambda_bb]]
# eta = [eta_a, eta_b]
# Marginal information parameters for x_a:
# Lambda_a = Lambda_aa - Lambda_ab Lambda_bb^{-1} Lambda_ba (Schur complement)
# eta_a = eta_a - Lambda_ab Lambda_bb^{-1} eta_b
GammaGaussian (joint Gaussian-Gamma):
# Joint distribution over x (Gaussian) and tau (precision):
# p(x, tau) = Gamma(tau | alpha, beta) * Normal(x | mu, (tau * Lambda)^{-1})
# This is represented as:
# log p(x, tau) = (alpha - 1) * log(tau) - beta * tau
# - 0.5 * tau * x^T Lambda x + tau * eta^T x + ...
# The tau variable acts as a global scale on the precision
# Marginalizing over tau yields a multivariate Student-t distribution for x