Implementation:Facebookresearch Habitat lab Benchmark evaluate

Knowledge Sources	Habitat-Lab
Domains	Evaluation, Embodied_AI
Last Updated	2026-02-15 02:00 GMT

Overview

Concrete evaluation method that runs an agent through Habitat episodes and returns aggregated performance metrics.

Description

The Benchmark.evaluate method dispatches to `local_evaluate` (single-process) or `remote_evaluate` (Habitat Challenge API). The local evaluation loop iterates episodes, runs the agent reset/act cycle, collects per-episode metrics, and returns a dict of mean values. A tqdm progress bar tracks evaluation progress.

Usage

Call on a Benchmark instance with an Agent and optional episode count. Returns a dict of aggregated metrics.

Code Reference

Source Location

• Repository: habitat-lab
• File: habitat-lab/habitat/core/benchmark.py
• Lines: L173-187 (evaluate), L124-171 (local_evaluate)

Signature

class Benchmark:
    def evaluate(
        self,
        agent: Agent,
        num_episodes: Optional[int] = None,
    ) -> Dict[str, float]:
        """
        Evaluate agent on episodes.

        Args:
            agent: Agent implementing reset() and act()
            num_episodes: Number of episodes to evaluate (None = all)
        Returns:
            Dict of metric name -> mean value across episodes
        """

Import

from habitat.core.benchmark import Benchmark

I/O Contract

Inputs

Name	Type	Required	Description
agent	Agent	Yes	Agent with reset() and act() methods
num_episodes	Optional[int]	No	Episodes to evaluate (None = entire dataset)

Outputs

Name	Type	Description
metrics	Dict[str, float]	Aggregated metrics: distance_to_goal, success, spl, soft_spl, etc.

Usage Examples

Evaluate Agent

from habitat.core.benchmark import Benchmark
from habitat_baselines.agents.simple_agents import GoalFollower

benchmark = Benchmark("habitat/config/benchmark/nav/pointnav/pointnav_habitat_test.yaml")
agent = GoalFollower(success_distance=0.2, goal_sensor_uuid="pointgoal_with_gps_compass")

metrics = benchmark.evaluate(agent, num_episodes=10)
for key, value in metrics.items():
    print(f"{key}: {value:.4f}")

Evaluate PPO Agent

from habitat_baselines.agents.ppo_agents import PPOAgent

agent = PPOAgent(config)
metrics = benchmark.evaluate(agent)
print(f"SPL: {metrics['spl']:.4f}, Success: {metrics['success']:.4f}")

Related Pages

Implements Principle

• Principle:Facebookresearch_Habitat_lab_Evaluation_Execution

Requires Environment

• Environment:Facebookresearch_Habitat_lab_Python_3_9_Core_Dependencies