Principle:Scikit learn Scikit learn Bayesian Regression

Knowledge Sources	Scikit_learn Scikit-learn Docs
Domains	Supervised Learning, Bayesian Inference
Last Updated	2026-02-08 15:00 GMT

Overview

Bayesian regression formulates linear regression within a probabilistic framework, placing prior distributions on model parameters and computing posterior distributions conditioned on observed data.

Description

Bayesian regression treats model parameters as random variables rather than fixed unknowns, incorporating prior beliefs about their distributions before observing data. The posterior distribution over parameters provides not only point estimates but also measures of uncertainty in predictions. This framework naturally handles regularization (priors act as regularizers) and provides principled uncertainty quantification. Bayesian regression sits within the broader family of probabilistic machine learning methods and is particularly valuable when uncertainty estimates are as important as the predictions themselves.

Usage

Use Bayesian regression when you need uncertainty estimates on predictions, when prior knowledge about parameter values is available, or when automatic regularization strength selection is desired. Bayesian Ridge regression is appropriate as a probabilistic alternative to Ridge regression, automatically tuning the regularization strength via evidence maximization. ARD (Automatic Relevance Determination) regression is preferred when many features are expected to be irrelevant, as it learns per-feature precision parameters that effectively prune irrelevant features, producing sparse solutions analogous to Lasso but within a Bayesian framework.

Theoretical Basis

Bayesian Linear Regression assumes the following generative model:

$y=X\beta +\epsilon ,\epsilon \sim {𝒩}(0,{\sigma }^{2}I)$

A conjugate prior is placed on the weights: $\beta \sim {𝒩}(0,{\alpha }^{-1}I)$.

The posterior distribution over weights is:

$p(\beta |X,y,\alpha ,{\sigma }^2)={𝒩}({\mu }_n,{\mathrm{\Sigma }}_n)$

where:

${\mathrm{\Sigma }}_n=(\alpha I+{\sigma }^{-2}{X}^{T}X{)}^{-1}$

${\mu }_n={\sigma }^{-2}{\mathrm{\Sigma }}_n{X}^{T}y$

Bayesian Ridge estimates the hyperparameters $\alpha$ (weight precision) and $\lambda$ (noise precision) by maximizing the marginal likelihood (evidence):

$p(y|X,\alpha ,\lambda )=\int p(y|X,\beta ,\lambda )p(\beta |\alpha )d\beta$

This integral is tractable in closed form for Gaussian priors and likelihoods. The hyperparameters are iteratively updated using the evidence framework (type-II maximum likelihood).

ARD Regression (Automatic Relevance Determination) generalizes Bayesian Ridge by assigning a separate precision parameter ${\alpha }_i$ to each weight:

${\beta }_i\sim {𝒩}(0,{\alpha }_i^{-1})$

During optimization, many ${\alpha }_i$ values diverge to infinity, driving the corresponding weights to zero. This produces a sparse model where only relevant features retain non-zero coefficients. The sparsity mechanism is sometimes called sparse Bayesian learning or the relevance vector machine framework.

Related Pages

• Implementation:Scikit_learn_Scikit_learn_BayesianRidge