Principle:Pyro ppl Pyro Posterior Predictive Analysis

Metadata

Field	Value
Principle ID	Pyro_ppl_Pyro_Posterior_Predictive_Analysis
Title	Posterior Predictive Analysis (SVI)
Project	Pyro (pyro-ppl/pyro)
Domains	Bayesian_Inference, Prediction
Implementation	Pyro_ppl_Pyro_Predictive_SVI
Repository	https://github.com/pyro-ppl/pyro

Summary

Posterior Predictive Analysis is the principle of generating predictions from the posterior distribution obtained through Stochastic Variational Inference (SVI). After training a model-guide pair, the posterior predictive distribution p(x_new | x_observed) is approximated by drawing latent samples from the trained guide q(z | x_observed) and running the model conditioned on those samples to produce predictions.

Motivation

The ultimate goal of Bayesian inference is not just to estimate parameters, but to make predictions that account for uncertainty. After training a model via SVI, the guide q(z) approximates the posterior p(z | data). The posterior predictive distribution integrates out the uncertainty in the latent variables:

p(x_new | x_observed) = integral of p(x_new | z) * q(z) dz

This distribution captures both:

• Aleatoric uncertainty -- inherent noise in the data-generating process (captured by p(x_new | z))
• Epistemic uncertainty -- uncertainty about the model parameters (captured by the spread of q(z))

By sampling from the posterior predictive, practitioners can:

• Generate realistic synthetic data
• Compute prediction intervals and credible regions
• Evaluate model fit via posterior predictive checks
• Make risk-aware decisions under uncertainty

Core Concepts

SVI-Based Posterior Predictive

In the SVI workflow, the posterior predictive is computed in two stages:

1. Sample from the guide: Draw N samples from the trained guide q(z). Each sample z_i represents a plausible set of latent variable values consistent with the observed data.
2. Run the model: For each guide sample z_i, run the model conditioned on z_i (using poutine.condition). The model produces observations and any other sample sites, generating one draw from p(x_new | z_i).

The collection of N model outputs approximates the posterior predictive distribution.

Sequential vs. Parallel Prediction

Posterior predictive samples can be generated in two ways:

• Sequential (parallel=False): The model is run N times in a loop, once per posterior sample. This is memory-efficient but slow.
• Parallel (parallel=True): The model is wrapped in an outer pyro.plate and run once with all N posterior samples batched together. This is fast but requires that the model correctly annotates all batch dimensions via pyro.plate.

Return Sites

By default, the Predictive class returns samples from all sample sites that are not in the posterior_samples (i.e., sites not sampled by the guide). This typically includes observed variables and any deterministic transformations. The return_sites parameter allows explicit control over which sites to include in the output.

How It Works

The SVI posterior predictive workflow:

1. The trained guide is called with the model's arguments to produce posterior_samples: a dict mapping site names to tensors of shape (num_samples, ...)
2. For each site in posterior_samples, the tensor is reshaped to align with the model's plate structure
3. The model is run, conditioned on the posterior samples via poutine.condition
4. Sample sites from the model trace are collected and reshaped to (num_samples, ...)
5. The resulting dict of tensors represents the posterior predictive samples

Example

import torch
import pyro
import pyro.distributions as dist
from pyro.infer import SVI, Trace_ELBO, Predictive
from pyro.optim import Adam
from pyro.infer.autoguide import AutoNormal

# Simple Bayesian linear regression
def model(x, y=None):
    weight = pyro.sample("weight", dist.Normal(0, 1))
    bias = pyro.sample("bias", dist.Normal(0, 1))
    sigma = pyro.sample("sigma", dist.LogNormal(0, 1))
    mean = weight * x + bias
    with pyro.plate("data", len(x)):
        pyro.sample("obs", dist.Normal(mean, sigma), obs=y)

# Train via SVI
guide = AutoNormal(model)
svi = SVI(model, guide, Adam({"lr": 0.01}), loss=Trace_ELBO())
x_train = torch.linspace(0, 1, 100)
y_train = 2.0 * x_train + 0.5 + 0.1 * torch.randn(100)

for step in range(1000):
    svi.step(x_train, y_train)

# Generate posterior predictive samples (SVI approach)
predictive = Predictive(model, guide=guide, num_samples=500,
                        return_sites=("obs",))
x_test = torch.linspace(0, 2, 50)
samples = predictive(x_test)
# samples["obs"] has shape (500, 50) -- 500 posterior predictive draws

# Compute prediction intervals
mean_prediction = samples["obs"].mean(dim=0)
lower = samples["obs"].quantile(0.025, dim=0)
upper = samples["obs"].quantile(0.975, dim=0)

Relationship to Other Principles

• Pyro_ppl_Pyro_MCMC_Posterior_Prediction -- The MCMC variant of posterior prediction uses explicit posterior samples instead of a guide. Both principles use the same Predictive class.
• Pyro_ppl_Pyro_Amortized_Variational_Inference -- When the guide is an amortized encoder (as in a VAE), posterior predictive analysis generates reconstructions and novel samples from the generative model.

Related Pages

Implemented By

• Implementation:Pyro_ppl_Pyro_Predictive_SVI

References

• Gelman, A. et al., "Bayesian Data Analysis", Chapter 6: Model Checking and Improvement
• Pyro Bayesian regression tutorial: https://pyro.ai/examples/bayesian_regression.html