Implementation:Online ml River Evaluate Progressive Val Score Anomaly

Knowledge Sources	River River Docs Beating the Hold-Out: Bounds for K-fold and Progressive Cross-Validation
Domains	Online Machine Learning, Anomaly Detection, Model Evaluation, Progressive Validation
Last Updated	2026-02-08 16:00 GMT

Overview

Concrete reference for using evaluate.progressive_val_score with anomaly detection models in the River library, documenting the anomaly-specific code path that uses score_one instead of predict_one for streaming evaluation.

Description

This is an angle-specific doc covering evaluate.progressive_val_score when used with AnomalyDetector or AnomalyFilter models. While the function signature is the same as for classification or regression evaluation, the internal behavior differs significantly for anomaly detectors.

The function detects anomaly models via utils.inspect.isanomalydetector(model) and utils.inspect.isanomalyfilter(model). When an anomaly model is detected:

The prediction function is set to model.score_one instead of model.predict_one or model.predict_proba_one.
For AnomalyFilter models, the raw score is additionally passed through model.classify(score) to produce a binary label.

This means the same progressive_val_score function works seamlessly with both classifiers and anomaly detectors, abstracting the interface difference internally.

The recommended metric for anomaly detection evaluation is metrics.ROCAUC, which evaluates the ranking quality of anomaly scores across all possible thresholds.

Usage

Use evaluate.progressive_val_score with anomaly detectors when:

You want a one-liner for evaluating an anomaly detector on a streaming dataset
You need periodic progress reporting (via print_every)
You want to evaluate with delayed labels (via delay)
You are comparing anomaly detectors on benchmark datasets

Code Reference

Source Location

river/evaluate/progressive_validation.py, lines 231-409 (progressive_val_score function), with anomaly-specific detection logic at lines 29-30 (within _progressive_validation).

Signature

def progressive_val_score(
    dataset: base.typing.Dataset,
    model,
    metric: metrics.base.Metric,
    moment: str | typing.Callable | None = None,
    delay: str | int | dt.timedelta | typing.Callable | None = None,
    print_every=0,
    show_time=False,
    show_memory=False,
    **print_kwargs,
) -> metrics.base.Metric:

Import

from river import evaluate
result = evaluate.progressive_val_score(
    dataset=dataset,
    model=model,
    metric=metric,
    print_every=1000
)

Parameters

Parameter	Type	Default	Description
dataset	Dataset	(required)	The stream of (x, y) observations to evaluate against.
model	Estimator	(required)	The model to evaluate. Anomaly detectors and filters are automatically detected.
metric	Metric	(required)	The metric for evaluation. `metrics.ROCAUC` is recommended for anomaly detection.
moment	str, callable, or None	None	Attribute or function for timestamping observations. None = arrival order.
delay	str, int, timedelta, callable, or None	None	Delay before revealing ground truth. None = no delay (standard progressive validation).
print_every	int	0	Print the current metric every N predictions. 0 = no printing.
show_time	bool	False	Display elapsed time in progress reports.
show_memory	bool	False	Display model memory usage in progress reports.

Anomaly Detection Logic

The key anomaly-specific code path within _progressive_validation:

# Line 29-30: Anomaly detector detection
if utils.inspect.isanomalydetector(model) or utils.inspect.isanomalyfilter(model):
    pred_func = model.score_one

# Line 74-75: AnomalyFilter classification
if utils.inspect.isanomalyfilter(model):
    y_pred = model.classify(y_pred)

I/O Contract

Inputs

Parameter	Type	Description
dataset	Iterable of (dict, int)	Stream of (features, label) pairs. Labels should be 0 (normal) or 1 (anomaly).
model	AnomalyDetector or AnomalyFilter	Any model implementing `score_one` and `learn_one`.
metric	Metric	A metric compatible with the model. `metrics.ROCAUC` for anomaly detectors; `metrics.ClassificationReport` for anomaly filters.

Outputs

Output	Type	Description
metric	metrics.base.Metric	The final metric object, updated with all observations. Print or access `.get()` for the numeric value.
(side effect)	printed output	Progress lines at intervals specified by `print_every`, e.g., `[1,000] ROCAUC: 88.43%`.

Usage Examples

Evaluating HalfSpaceTrees with ROCAUC:

from river import anomaly, compose, datasets, evaluate, metrics, preprocessing

model = compose.Pipeline(
    preprocessing.MinMaxScaler(),
    anomaly.HalfSpaceTrees(seed=42)
)

evaluate.progressive_val_score(
    dataset=datasets.CreditCard().take(2500),
    model=model,
    metric=metrics.ROCAUC(),
    print_every=1000
)
# [1,000] ROCAUC: 88.43%
# [2,000] ROCAUC: 89.28%
# [2,500] ROCAUC: 91.15%
# ROCAUC: 91.15%

Evaluating OneClassSVM with QuantileFilter:

from river import anomaly, datasets, evaluate, metrics

model = anomaly.QuantileFilter(
    anomaly.OneClassSVM(nu=0.2),
    q=0.995
)

evaluate.progressive_val_score(
    dataset=datasets.CreditCard().take(2500),
    model=model,
    metric=metrics.ROCAUC(),
    print_every=1000
)
# [1,000] ROCAUC: 74.40%
# [2,000] ROCAUC: 74.60%
# [2,500] ROCAUC: 74.68%
# ROCAUC: 74.68%

Equivalent manual evaluation loop:

from river import anomaly, compose, datasets, metrics, preprocessing

model = compose.Pipeline(
    preprocessing.MinMaxScaler(),
    anomaly.HalfSpaceTrees(seed=42)
)

metric = metrics.ROCAUC()

for x, y in datasets.CreditCard().take(2500):
    score = model.score_one(x)   # score_one, not predict_one
    metric.update(y, score)
    model.learn_one(x)

print(metric)
# ROCAUC: 91.15%

With time and memory tracking:

from river import anomaly, compose, datasets, evaluate, metrics, preprocessing

model = compose.Pipeline(
    preprocessing.MinMaxScaler(),
    anomaly.HalfSpaceTrees(seed=42)
)

evaluate.progressive_val_score(
    dataset=datasets.CreditCard().take(5000),
    model=model,
    metric=metrics.ROCAUC(),
    print_every=2500,
    show_time=True,
    show_memory=True
)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment