Principle:Pyro ppl Pyro Mixture Models
| Knowledge Sources | |
|---|---|
| Domains | Mixture Models, Density Estimation, Clustering |
| Last Updated | 2026-02-09 09:00 GMT |
Overview
Mixture models represent complex, multimodal probability distributions as weighted combinations of simpler component distributions, enabling flexible density estimation and unsupervised clustering.
Description
A mixture model expresses a probability distribution as a convex combination of K component distributions:
p(x) = sum_{k=1}^{K} pi_k * f_k(x)
where pi_k are non-negative mixing weights that sum to 1, and f_k are the component densities.
Mixture of Diagonal Normals: Each component is a multivariate Gaussian with a diagonal covariance matrix. This is computationally efficient while still able to approximate complex multimodal distributions. The diagonal constraint means each dimension is independent within a component, but the mixture as a whole can capture complex correlations.
Gaussian Scale Mixture (GSM): A construction where a random variable X is represented as X = sqrt(W) * Z, where Z is standard Gaussian and W is a non-negative random variable (the mixing variable). Different choices of the distribution of W yield different marginal distributions for X:
- W ~ InverseGamma gives Student's t-distribution
- W ~ point mass gives Gaussian
- W ~ Exponential gives Laplace distribution
GSMs are important because they:
- Provide a unified framework for generating heavy-tailed distributions from Gaussian building blocks.
- Enable Gibbs sampling: conditioned on W, the distribution is Gaussian; conditioned on X, W has a known conditional.
- Are used in sparse signal processing and image denoising (sparse priors are often GSMs).
Usage
Use mixture models when:
- Data exhibits multiple distinct clusters or modes.
- You need a flexible nonparametric density estimator.
- Building variational approximations that are more expressive than single Gaussians.
- Modeling heterogeneous populations where different subgroups follow different distributions.
- Constructing heavy-tailed or sparse priors via the scale mixture construction.
Theoretical Basis
Finite mixture model:
# Mixture density
p(x | pi, theta) = sum_{k=1}^{K} pi_k * f(x | theta_k)
# Log-likelihood for N observations:
log p(X | pi, theta) = sum_{n=1}^{N} log(sum_{k=1}^{K} pi_k * f(x_n | theta_k))
# Latent variable formulation:
# z_n ~ Categorical(pi) # component assignment
# x_n | z_n=k ~ f(x | theta_k) # observation from component k
Mixture of Diagonal Normals:
# K components, D dimensions
# Parameters: pi (K,), mu (K, D), sigma (K, D)
p(x) = sum_{k=1}^{K} pi_k * product_{d=1}^{D} Normal(x_d | mu_{k,d}, sigma_{k,d}^2)
# log_prob(x) = logsumexp_k(log(pi_k) + sum_d log Normal(x_d | mu_{k,d}, sigma_{k,d}^2))
Gaussian Scale Mixture:
# Scale mixture representation:
# X = sqrt(W) * Z where Z ~ Normal(0, Sigma), W ~ p(w)
# Marginal density:
p(x) = integral_0^inf Normal(x | 0, w * Sigma) * p(w) dw
# Special cases:
# p(w) = InverseGamma(nu/2, nu/2) => X ~ StudentT(nu, 0, Sigma)
# p(w) = delta(1) => X ~ Normal(0, Sigma)
# p(w) = Exponential(1/2) => X ~ Laplace(0, Sigma)
# Conditional structure (enables Gibbs sampling):
# W | X ~ some known distribution (depends on p(w))
# X | W ~ Normal(0, W * Sigma)
EM algorithm for mixture parameter estimation:
# E-step: compute responsibilities
r_{nk} = pi_k * f(x_n | theta_k) / sum_j pi_j * f(x_n | theta_j)
# M-step: update parameters
pi_k = (1/N) * sum_n r_{nk}
mu_k = sum_n r_{nk} * x_n / sum_n r_{nk}
sigma_k^2 = sum_n r_{nk} * (x_n - mu_k)^2 / sum_n r_{nk}