Implementation:Online ml River Bandit Evaluate

Knowledge Sources	Online_ml_River
Domains	Online_Learning, Multi_Armed_Bandits, Evaluation, Offline_Evaluation
Last Updated	2026-02-08 16:00 GMT

Overview

Utility functions for evaluating bandit policies both online (with Gymnasium environments) and offline (with historical data using replay methodology).

Description

The module provides two evaluation functions. The evaluate function benchmarks policies on Gymnasium environments by running multiple episodes and collecting step-by-step statistics. It creates independent copies of policies and environments for fair comparison. The evaluate_offline function performs off-policy evaluation using historical data through the replay methodology, where the policy's decision is compared against the historical decision, and rewards are only used when they match. This enables data-driven evaluation without environment interaction.

Usage

Use evaluate() for online testing with simulated environments and multiple policies. Use evaluate_offline() when you have logged data from a production system and want to estimate how a new policy would perform. The replay method is unbiased but requires sufficient overlap between the logging policy and evaluation policy.

Code Reference

Source Location

Repository: Online_ml_River
File: river/bandit/evaluate.py

Signature

def evaluate(
    policies: list[bandit.base.Policy],
    env: gym.Env,
    reward_stat: stats.base.Univariate | None = None,
    n_episodes: int = 20,
    seed: int | None = None,
):
    ...

def evaluate_offline(
    policy: bandit.base.Policy,
    history: History | bandit.datasets.BanditDataset,
    reward_stat: stats.base.Univariate | None = None,
) -> tuple[stats.base.Univariate, int]:
    ...

Import

from river import bandit

I/O Contract

Function	Inputs	Outputs
evaluate	policies, env, reward_stat, n_episodes, seed	Generator of step dictionaries
evaluate_offline	policy, history, reward_stat	(reward_stat, n_samples_used)

Usage Examples

import gymnasium as gym
import pandas as pd
from river import bandit

# Online evaluation
trace = bandit.evaluate(
    policies=[
        bandit.UCB(delta=1, seed=42),
        bandit.EpsilonGreedy(epsilon=0.1, seed=42),
    ],
    env=gym.make(
        'river_bandits/CandyCaneContest-v0',
        max_episode_steps=100
    ),
    n_episodes=5,
    seed=42
)

# Convert to DataFrame for analysis
trace_df = pd.DataFrame(trace)
print(trace_df.groupby('policy_idx')['reward'].sum())

# Offline evaluation with historical data
news = bandit.datasets.NewsArticles()
total_reward, n_samples_used = bandit.evaluate_offline(
    policy=bandit.RandomPolicy(seed=42),
    history=news,
)

print(f"Total reward: {total_reward}")
print(f"Samples used: {n_samples_used}")

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment