Principle:Farama Foundation Gymnasium Q Learning Tabular

Knowledge Sources	Q-Learning Sutton and Barto RL Gymnasium Tutorials
Domains	Reinforcement_Learning, Value_Based_Methods
Last Updated	2026-02-15 03:00 GMT

Overview

A model-free temporal-difference learning algorithm that estimates the optimal action-value function using a lookup table and an epsilon-greedy exploration strategy.

Description

Tabular Q-Learning maintains a table mapping state-action pairs to estimated Q-values. At each step, the agent:

1. Observes state $s$
2. Selects action via epsilon-greedy: random with probability $ϵ$, greedy otherwise
3. Receives reward $r$ and next state $s^{\prime }$
4. Updates $Q(s,a)$ using the TD update rule

Q-Learning is off-policy: the update uses $\underset{a}{\max }Q(s^{\prime },a)$ regardless of the action actually taken. This enables learning the optimal policy while exploring.

Requirements for tabular Q-Learning:

• Discrete state space: States must be hashable (tuples, integers)
• Discrete action space: Typically Discrete(n) in Gymnasium
• Sufficient exploration: Epsilon-greedy with decaying epsilon

Usage

Use tabular Q-Learning for environments with small, discrete state and action spaces (e.g., Blackjack, FrozenLake, Taxi). For continuous or high-dimensional states, use Deep Q-Networks (DQN) instead.

Theoretical Basis

The Q-Learning update rule: $Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha [r_{t+1}+\gamma \underset{a}{\max }Q(s_{t+1},a)-Q(s_t,a_t)]$

Where:

• $\alpha$: Learning rate
• $\gamma$: Discount factor
• $\underset{a}{\max }Q(s^{\prime },a)$: Off-policy bootstrap target

Convergence is guaranteed under the Robbins-Monro conditions: every state-action pair is visited infinitely often, and the learning rate decays appropriately.

Epsilon-greedy policy: $\pi (a|s)=\{\begin{array}{ll}1-ϵ+\frac{ϵ}{|A|}& \text{if }a=\arg \underset{a^{\prime }}{\max }Q(s,a^{\prime })\\ \frac{ϵ}{|A|}& \text{otherwise}\end{array}$

Related Pages

Implemented By

• Implementation:Farama_Foundation_Gymnasium_Q_Table_Agent