Principle:Pyro ppl Pyro Conditional Density Estimation
| Knowledge Sources | |
|---|---|
| Domains | Conditional Density Estimation, Amortized Inference, Neural Networks |
| Last Updated | 2026-02-09 09:00 GMT |
Overview
Conditional density estimation learns to output a full probability distribution over target variables as a function of conditioning variables, forming the basis of amortized inference in probabilistic programming.
Description
In standard density estimation, the goal is to learn p(x). In conditional density estimation, the goal is to learn p(y | x) -- a full probability distribution over y that varies as a function of the input x.
This is more general than regression (which learns E[y|x]) or classification (which learns P(class|x) for discrete classes). Conditional density estimation captures the full uncertainty and multimodality of the conditional distribution.
In the context of probabilistic programming, conditional density estimation is the foundation of amortized inference. Instead of running a separate optimization for each observed dataset x to find the posterior q(z|x), an amortized inference network learns a single function that maps any x to an approximate posterior distribution:
q(z | x; phi) = ConditionalDistribution(x; phi)
This network is trained once on many simulated (x, z) pairs and then reused at test time, providing instant approximate posteriors without per-dataset optimization.
The ConditionalDistribution abstraction represents this concept: it is a callable that takes conditioning variables as input and returns a distribution object. The distribution's parameters are computed by a neural network from the conditioning input.
Common architectures include:
- Conditional Gaussian: A neural network outputs the mean and covariance of a Gaussian, conditioned on input.
- Conditional mixture: A mixture density network outputs mixing weights, means, and variances of Gaussian components.
- Conditional normalizing flow: A neural network parameterizes the flow transforms, conditioned on input.
Usage
Use conditional density estimation when:
- Building amortized inference networks (guide networks) for variational inference.
- Learning proposal distributions for importance sampling that adapt to observed data.
- Performing simulation-based inference where the likelihood is implicit.
- Constructing flexible conditional priors in hierarchical models.
- Building generative models where the conditional generation process is complex and multimodal.
Theoretical Basis
Amortized variational inference:
# Standard VI: for each dataset x, optimize
# phi*(x) = argmax_phi ELBO(x; phi)
# Cost: O(optimization_steps) per dataset
# Amortized VI: learn a single network
# phi* = argmax_phi E_{p(x)}[ELBO(x; phi)]
# Cost: O(1) per dataset at test time (single forward pass)
# The guide q(z | x; phi) is a conditional distribution
# parameterized by a neural network with weights phi
Conditional distribution interface:
# Abstract interface:
class ConditionalDistribution:
def condition(x) -> Distribution:
# Returns a distribution object whose parameters
# are computed from x via a neural network
params = neural_network(x)
return Distribution(params)
# Example: Conditional Gaussian
class ConditionalGaussian(ConditionalDistribution):
def condition(x):
mu, log_sigma = split(neural_net(x))
return Normal(mu, exp(log_sigma))
Training objective (amortized ELBO):
# For each training example x_i:
# 1. Compute conditional distribution: q_i = guide.condition(x_i)
# 2. Sample: z_i ~ q_i
# 3. Compute ELBO_i = log p(x_i, z_i) - log q_i(z_i)
# Optimize: max_phi (1/N) * sum_i ELBO_i
# Gradients flow through the conditional distribution parameters
# back to the neural network weights phi
Mixture density networks as a specific conditional density estimator:
# Output K Gaussian components conditioned on x:
# pi(x), mu_1(x)...mu_K(x), sigma_1(x)...sigma_K(x) = neural_net(x)
# Conditional density:
# p(y | x) = sum_k pi_k(x) * Normal(y | mu_k(x), sigma_k(x)^2)