Implementation:Online ml River Bandit ThompsonSampling
| Knowledge Sources | |
|---|---|
| Domains | Online_Learning, Multi_Armed_Bandits, Bayesian_Methods, Probabilistic_Programming |
| Last Updated | 2026-02-08 16:00 GMT |
Overview
A Bayesian bandit algorithm that uses probability matching by sampling from posterior distributions to balance exploration and exploitation.
Description
Thompson Sampling is a probabilistic approach where each arm maintains a posterior distribution over its reward. At each step, the algorithm samples a value from each arm's posterior and selects the arm with the highest sample. This naturally balances exploration (sampling from uncertain distributions) and exploitation (favoring arms with high expected rewards). While Beta distributions are commonly used for binary rewards, any probability distribution can be used depending on the reward structure. The algorithm has strong theoretical guarantees and often performs well in practice.
Usage
Use Thompson Sampling when you want a principled Bayesian approach with good empirical performance. It's particularly effective with Beta distributions for binary rewards or Gaussian distributions for continuous rewards. The algorithm adapts naturally to the uncertainty in reward estimates without needing tuning parameters.
Code Reference
Source Location
- Repository: Online_ml_River
- File: river/bandit/thompson.py
Signature
class ThompsonSampling(bandit.base.Policy):
def __init__(
self,
reward_obj: proba.base.Distribution | None = None,
burn_in=0,
seed: int | None = None,
):
...
Import
from river import bandit
I/O Contract
| Parameter | Type | Description |
|---|---|---|
| reward_obj | proba.Distribution (optional) | Distribution to sample from (defaults to proba.Beta()) |
| burn_in | int (default: 0) | Minimum pulls per arm before sampling |
| seed | int (optional) | Random seed for reproducibility |
Usage Examples
import gymnasium as gym
from river import bandit
from river import proba
from river import stats
env = gym.make('river_bandits/CandyCaneContest-v0')
_ = env.reset(seed=42)
_ = env.action_space.seed(123)
# Use Beta distribution for binary rewards
policy = bandit.ThompsonSampling(
reward_obj=proba.Beta(),
seed=101
)
metric = stats.Sum()
while True:
arm = policy.pull(range(env.action_space.n))
observation, reward, terminated, truncated, info = env.step(arm)
policy.update(arm, reward)
metric.update(reward)
if terminated or truncated:
break
print(metric) # Sum: 820.
# Use Gaussian for continuous rewards
policy_gauss = bandit.ThompsonSampling(
reward_obj=proba.Gaussian(),
seed=42
)