Implementation:Online ml River Bandit Evaluate
| Knowledge Sources | |
|---|---|
| Domains | Online_Learning, Multi_Armed_Bandits, Evaluation, Offline_Evaluation |
| Last Updated | 2026-02-08 16:00 GMT |
Overview
Utility functions for evaluating bandit policies both online (with Gymnasium environments) and offline (with historical data using replay methodology).
Description
The module provides two evaluation functions. The evaluate function benchmarks policies on Gymnasium environments by running multiple episodes and collecting step-by-step statistics. It creates independent copies of policies and environments for fair comparison. The evaluate_offline function performs off-policy evaluation using historical data through the replay methodology, where the policy's decision is compared against the historical decision, and rewards are only used when they match. This enables data-driven evaluation without environment interaction.
Usage
Use evaluate() for online testing with simulated environments and multiple policies. Use evaluate_offline() when you have logged data from a production system and want to estimate how a new policy would perform. The replay method is unbiased but requires sufficient overlap between the logging policy and evaluation policy.
Code Reference
Source Location
- Repository: Online_ml_River
- File: river/bandit/evaluate.py
Signature
def evaluate(
policies: list[bandit.base.Policy],
env: gym.Env,
reward_stat: stats.base.Univariate | None = None,
n_episodes: int = 20,
seed: int | None = None,
):
...
def evaluate_offline(
policy: bandit.base.Policy,
history: History | bandit.datasets.BanditDataset,
reward_stat: stats.base.Univariate | None = None,
) -> tuple[stats.base.Univariate, int]:
...
Import
from river import bandit
I/O Contract
| Function | Inputs | Outputs |
|---|---|---|
| evaluate | policies, env, reward_stat, n_episodes, seed | Generator of step dictionaries |
| evaluate_offline | policy, history, reward_stat | (reward_stat, n_samples_used) |
Usage Examples
import gymnasium as gym
import pandas as pd
from river import bandit
# Online evaluation
trace = bandit.evaluate(
policies=[
bandit.UCB(delta=1, seed=42),
bandit.EpsilonGreedy(epsilon=0.1, seed=42),
],
env=gym.make(
'river_bandits/CandyCaneContest-v0',
max_episode_steps=100
),
n_episodes=5,
seed=42
)
# Convert to DataFrame for analysis
trace_df = pd.DataFrame(trace)
print(trace_df.groupby('policy_idx')['reward'].sum())
# Offline evaluation with historical data
news = bandit.datasets.NewsArticles()
total_reward, n_samples_used = bandit.evaluate_offline(
policy=bandit.RandomPolicy(seed=42),
history=news,
)
print(f"Total reward: {total_reward}")
print(f"Samples used: {n_samples_used}")