Workflow:Farama Foundation Gymnasium Agent Evaluation Recording
| Knowledge Sources | |
|---|---|
| Domains | Reinforcement_Learning, Evaluation, Visualization |
| Last Updated | 2026-02-15 03:00 GMT |
Overview
End-to-end process for recording, evaluating, and visualizing the performance of a trained reinforcement learning agent using Gymnasium's recording and statistics wrappers.
Description
This workflow covers the standard procedure for evaluating a trained RL agent and capturing its behavior through video recordings and performance statistics. It leverages Gymnasium's RecordVideo wrapper for capturing MP4 video files of agent episodes and RecordEpisodeStatistics wrapper for tracking numerical metrics (episode rewards, lengths, and wall-clock times). The workflow supports two recording modes: full recording during evaluation (every episode captured) and periodic recording during training (every Nth episode). It also covers computing summary statistics, generating learning curve plots, and optionally integrating with experiment tracking tools like Weights and Biases.
Usage
Execute this workflow when you have a trained agent and need to assess its performance quantitatively and visually. This applies when you want to create demonstration videos, generate evaluation metrics for comparison across algorithms, debug agent behavior by inspecting specific episodes, track learning progress during training, or prepare results for publication or presentation.
Execution Steps
Step 1: Evaluation Environment Setup
Create the evaluation environment with render_mode="rgb_array" to enable video frame capture. Apply the RecordVideo wrapper with configuration for the video output folder, filename prefix, and episode trigger function (lambda that controls which episodes are recorded). Apply RecordEpisodeStatistics wrapper with an appropriate buffer_length matching the number of evaluation episodes.
Key considerations:
- render_mode must be "rgb_array" for RecordVideo to capture frames
- The episode_trigger function receives the episode index and returns True/False
- For full evaluation recording, use episode_trigger=lambda x: True
- For periodic training recording, use episode_trigger=lambda x: x % period == 0
- Wrapper order matters: apply RecordVideo before RecordEpisodeStatistics
Step 2: Evaluation Episode Execution
Run the trained agent through multiple evaluation episodes with exploration disabled (epsilon set to zero or using deterministic action selection). For each episode, reset the environment, execute the agent's policy until termination or truncation, and track per-episode rewards. The RecordVideo wrapper automatically saves MP4 files and RecordEpisodeStatistics automatically records metrics.
Key considerations:
- Disable exploration (epsilon = 0) during evaluation to test the pure learned policy
- Run sufficient episodes (typically 4-100) for statistically meaningful results
- Each recorded episode produces a separate MP4 file in the specified video folder
- Episode statistics are accessible via env.return_queue, env.length_queue, and env.time_queue
Step 3: Performance Metrics Computation
Extract episode statistics from the wrapper queues and compute summary metrics including mean reward, standard deviation, win rate (fraction of episodes exceeding a threshold), and average episode length. These metrics provide a quantitative assessment of agent quality and consistency.
Key considerations:
- Access statistics via env.return_queue, env.length_queue, and env.time_queue
- Compute confidence intervals for statistically rigorous comparisons
- Compare metrics against known baselines (random policy, optimal policy if known)
- Track both mean performance and variance to assess consistency
Step 4: Learning Curve Visualization
Generate plots showing the agent's learning progress over training. Use moving averages to smooth noisy per-episode data and reveal overall trends. Create multi-panel figures showing episode rewards, episode lengths, and optionally training loss or error metrics. Use matplotlib for creating publication-quality figures.
Key considerations:
- Choose an appropriate rolling window size (e.g., 100-500 episodes) for smoothing
- Plot raw data with low alpha transparency alongside the moving average
- Include axis labels, titles, and legends for clarity
- Save figures to disk for inclusion in reports or papers
Step 5: Experiment Tracking Integration
Optionally integrate recording with experiment tracking tools (such as Weights and Biases) to log metrics, upload videos, and compare across training runs. Log per-episode statistics during training and upload recorded videos at milestone episodes for remote monitoring and team collaboration.
Key considerations:
- Initialize experiment tracking before the training loop begins
- Log episode metrics (reward, length, time) after each episode completion
- Upload video files at recording intervals for visual progress tracking
- Use experiment tags and grouping for organizing multiple runs