Principle:Online ml River Conformal Prediction

Knowledge Sources	Algorithmic Learning in a Random World A Tutorial on Conformal Prediction
Domains	Online_Learning Uncertainty_Quantification Regression
Last Updated	2026-02-08 18:00 GMT

Overview

Conformal prediction is a distribution-free framework for constructing prediction intervals (or prediction sets) with finite-sample coverage guarantees. Given a desired confidence level $1 - α$ , conformal prediction produces intervals that contain the true target value with probability at least $1 - α$ , without requiring assumptions about the data distribution.

Description

Standard regression models produce point predictions but do not quantify the uncertainty of those predictions. Conformal prediction addresses this by wrapping any point predictor with a procedure that outputs prediction intervals with guaranteed coverage.

The key idea is to use nonconformity scores -- measures of how unusual a new observation is relative to past data -- to calibrate the width of prediction intervals. The most common approach for regression is:

Split conformal: Hold out a calibration set, compute residuals on it, and use their quantiles to set interval widths.
Full conformal: Refit the model for each candidate target value (computationally expensive but exact).
Jackknife (leave-one-out): Use leave-one-out residuals to calibrate intervals, balancing computational cost with statistical efficiency.
Jackknife+: A refinement that provides exact finite-sample coverage guarantees even with data reuse.

In the online (streaming) setting, conformal prediction can be adapted by maintaining a rolling window of recent residuals and updating the interval calibration incrementally.

Usage

Use conformal prediction when:

You need prediction intervals with formal coverage guarantees.
You want uncertainty quantification without distributional assumptions.
You need to communicate prediction reliability to downstream decision systems.
You want a method that works with any underlying regression model (model-agnostic).

Theoretical Basis

Nonconformity score: For a regression model $f$ , the residual-based nonconformity score is:

s_i = |y_i - f(x_i)|

Split conformal interval: Given calibration residuals $s_{1}, \dots, s_{n}$ and significance level $α$ :

q = quantile(s_1, ..., s_n, level = ceil((n+1)(1-alpha)) / n)
Interval(x_new) = [f(x_new) - q,  f(x_new) + q]

Coverage guarantee (exchangeability): If the data points $(x_{1}, y_{1}), \dots, (x_{n + 1}, y_{n + 1})$ are exchangeable, then:

P(y_{n+1} in Interval(x_{n+1})) >= 1 - alpha

This holds for any underlying model $f$ and any data distribution.

Jackknife prediction interval: Let $f_{- i}$ be the model trained without observation $i$ , and $r_{i} = | y_{i} - f_{- i} (x_{i}) |$ :

Interval(x_new) = [f(x_new) - q_jack,  f(x_new) + q_jack]
where q_jack = quantile(r_1, ..., r_n, level = 1 - alpha)

Online adaptation: In streaming, residuals are computed progressively and a sliding window of the most recent $n$ residuals is maintained. The quantile $q$ is updated incrementally as new residuals enter and old ones exit the window.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment