Principle:Scikit learn Scikit learn Gaussian Process

Knowledge Sources	Scikit_learn Scikit-learn Docs
Domains	Supervised Learning, Bayesian Inference
Last Updated	2026-02-08 15:00 GMT

Overview

Gaussian processes define a distribution over functions, providing a non-parametric Bayesian approach to regression and classification with built-in uncertainty quantification.

Description

A Gaussian Process (GP) is a collection of random variables, any finite number of which follow a joint Gaussian distribution. GPs are fully specified by a mean function and a covariance (kernel) function, which encodes assumptions about the function being modeled (smoothness, periodicity, length scale). They solve the problem of making predictions with well-calibrated uncertainty estimates without committing to a fixed parametric form. GPs are particularly valuable in settings where uncertainty quantification is critical, such as Bayesian optimization, active learning, and safety-critical applications.

Usage

Use Gaussian Process Regression (GPR) when you need both predictions and uncertainty estimates, when the dataset is small to moderate in size (GPs scale as $O(n^3)$), and when the function is expected to be smooth. Use Gaussian Process Classification (GPC) for probabilistic classification with uncertainty estimates. The choice of kernel is critical: use the RBF kernel for smooth functions, the Matern kernel for functions with varying smoothness, periodic kernels for periodic patterns, and composite kernels (sums and products) for complex structure. GPs are not suitable for very large datasets without approximation methods.

Theoretical Basis

Gaussian Process is defined as:

$f(x)\sim {𝒢}{𝒫}(m(x),k(x,x^{\prime }))$

where $m(x)$ is the mean function (often zero) and $k(x,x^{\prime })$ is the covariance (kernel) function.

Gaussian Process Regression: Given training data $(X,y)$ with noise model $y=f(x)+\epsilon$, $\epsilon \sim {𝒩}(0,{\sigma }_n^2)$, the predictive distribution at test points $X_{*}$ is:

$f_{*}|X,y,X_{*}\sim {𝒩}({\overline{f}}_{*},\text{cov}(f_{*}))$

where: ${\overline{f}}_{*}=K(X_{*},X)[K(X,X)+{\sigma }_n^{2}I{]}^{-1}y$ $\text{cov}(f_{*})=K(X_{*},X_{*})-K(X_{*},X)[K(X,X)+{\sigma }_n^{2}I{]}^{-1}K(X,X_{*})$

Gaussian Process Classification: For binary classification, the latent function is passed through a sigmoid (or probit) link function:

$p(y=1|x)=\sigma (f(x))$

Since the posterior over $f$ is no longer Gaussian, approximate inference is needed (Laplace approximation or Expectation Propagation).

Common Kernel Functions:

RBF (Squared Exponential): $k(x,x^{\prime })={\sigma }_f^2\exp (-\frac{\Vert x-x^{\prime }{\Vert }^2}{2{\ell }^2})$

Matern kernel: $k(x,x^{\prime })={\sigma }_f^2\frac{2^{1-\nu }}{\mathrm{\Gamma }(\nu )}{\left(\frac{\sqrt{2\nu }\Vert x-x^{\prime }\Vert }{\ell }\right)}^{\nu }{K}_{\nu }\left(\frac{\sqrt{2\nu }\Vert x-x^{\prime }\Vert }{\ell }\right)$

where $K_{\nu }$ is the modified Bessel function. The parameter $\nu$ controls smoothness.

Rational Quadratic: $k(x,x^{\prime })={\sigma }_f^2{(1+\frac{\Vert x-x^{\prime }{\Vert }^2}{2\alpha {\ell }^2})}^{-\alpha }$

Hyperparameter optimization is performed by maximizing the log marginal likelihood:

$\log p(y|X,\theta )=-\frac{1}{2}{y}^T(K+{\sigma }_n^{2}I{)}^{-1}y-\frac{1}{2}\log |K+{\sigma }_n^{2}I|-\frac{n}{2}\log 2\pi$

This provides a principled, Bayesian approach to model selection that automatically balances data fit and model complexity (Occam's razor).

Related Pages

• Implementation:Scikit_learn_Scikit_learn_GaussianProcessClassifier
• Implementation:Scikit_learn_Scikit_learn_GaussianProcessRegressor
• Implementation:Scikit_learn_Scikit_learn_GPKernels