Principle:Pyro ppl Pyro Bayesian Optimization

Knowledge Sources	A Tutorial on Bayesian Optimization Practical Bayesian Optimization of Machine Learning Algorithms Gaussian Processes for Machine Learning
Domains	Bayesian Optimization, Gaussian Processes, Black-Box Optimization
Last Updated	2026-02-09 09:00 GMT

Overview

Bayesian optimization uses a Gaussian process surrogate model and an acquisition function to efficiently optimize expensive black-box functions by balancing exploration of uncertain regions with exploitation of promising regions.

Description

Many optimization problems involve functions that are expensive to evaluate: training a neural network with specific hyperparameters, running a physical experiment, or simulating a complex system. Bayesian optimization (BO) addresses this by building a cheap-to-evaluate surrogate model of the objective function and using it to decide where to evaluate next.

The BO loop consists of:

1. Surrogate model: A Gaussian process (GP) is fit to all previous evaluations. The GP provides both a mean prediction (best guess of the function value) and uncertainty (how confident we are).

1. Acquisition function: A function that uses the GP's predictions to score candidate evaluation points. Good acquisition functions balance:

• Exploitation: Evaluating near the current best point (where the GP mean is high).
• Exploration: Evaluating where uncertainty is high (where the GP variance is large).

1. Optimization: The acquisition function is optimized (cheaply, since it only queries the GP) to find the next evaluation point.

1. Evaluation: The expensive objective function is evaluated at the chosen point.

1. Update: The GP is updated with the new observation, and the process repeats.

Common acquisition functions include:

• Expected Improvement (EI): The expected amount by which the next evaluation will improve over the current best.
• Upper Confidence Bound (UCB): Mean + beta * standard deviation, where beta controls the exploration-exploitation trade-off.
• Probability of Improvement (PI): The probability that the next evaluation exceeds the current best.

Usage

Use Bayesian optimization when:

• The objective function is expensive to evaluate (minutes to hours per evaluation).
• Gradient information is unavailable (black-box optimization).
• The input space is low-to-moderate dimensional (typically d < 20).
• You need sample-efficient optimization (minimize the number of evaluations).
• Hyperparameter tuning for machine learning models.
• Design of experiments in science and engineering.

Theoretical Basis

GP surrogate model:

# Given observations D = {(x_i, y_i)}_{i=1}^n:
# GP posterior at candidate point x:
# mu(x) = k(x, X) [K(X, X) + sigma^2 I]^{-1} y
# sigma^2(x) = k(x, x) - k(x, X) [K(X, X) + sigma^2 I]^{-1} k(X, x)

# mu(x): best estimate of f(x)
# sigma(x): uncertainty about f(x)

Expected Improvement (EI):

# Current best: y_best = max(y_1, ..., y_n)
# Improvement: I(x) = max(f(x) - y_best, 0)

# Expected Improvement:
# EI(x) = E[I(x) | D]
#        = (mu(x) - y_best) * Phi(z) + sigma(x) * phi(z)
# where z = (mu(x) - y_best) / sigma(x)
# Phi: standard normal CDF
# phi: standard normal PDF

# Properties:
# - EI = 0 at observed points (no improvement expected at known locations)
# - EI > 0 where mu > y_best (exploitation) or sigma is large (exploration)
# - EI has a closed-form expression (cheap to evaluate)

Upper Confidence Bound (UCB):

# UCB(x) = mu(x) + beta * sigma(x)

# beta: exploration parameter
# beta_t = 2 * log(|D| * t^2 * pi^2 / 6 / delta)  (theoretical schedule)
# In practice: beta = 2.0 works well

# Theoretical guarantee (GP-UCB):
# Cumulative regret after T iterations:
# R_T = O(sqrt(T * beta_T * gamma_T))
# where gamma_T is the maximum information gain of the kernel

Full BO loop:

# Initialize: evaluate at n_init random points
# D = {(x_i, y_i) for i = 1, ..., n_init}

# For t = n_init + 1, ..., budget:
#   1. Fit GP to D:
#      theta* = argmax_theta log p(y | X, theta)  # optimize hyperparameters
#      gp = GP(X, y, kernel(theta*))
#
#   2. Find next point:
#      x_next = argmax_x acquisition(x | gp)
#      (optimize acquisition function, e.g., using L-BFGS with random restarts)
#
#   3. Evaluate:
#      y_next = f(x_next)  # expensive evaluation
#
#   4. Update:
#      D = D union {(x_next, y_next)}

# Return: argmax_{x in D} y

Related Pages

• Implementation:Pyro_ppl_Pyro_GP_BayesOpt