Principle:EvolvingLMMs Lab Lmms eval Task Directory Structure

Knowledge Sources	lmms-eval
Domains	Task_Management, Configuration
Last Updated	2026-02-14 00:00 GMT

Overview

Evaluation tasks must be organized in a discoverable directory structure so that the framework can automatically index and load them by name.

Description

A well-defined directory structure is essential for a scalable evaluation framework. When hundreds of benchmarks exist, a convention-based approach to task discovery avoids manual registration and makes it easy for contributors to add new tasks without modifying core framework code.

In lmms-eval, each task occupies a named directory under a root tasks folder. The directory contains at minimum a YAML configuration file that declares the task's properties (dataset path, prompt templates, metrics, etc.) and optionally a Python utility module (utils.py) that provides custom functions referenced by the YAML.

The standard directory layout is:

lmms_eval/tasks/
    {task_name}/
        {task_name}.yaml      # Task configuration (required)
        utils.py               # Utility functions (optional)
        _default_template.yaml # Shared base config (optional)

The framework walks this directory tree at startup, reading each YAML file and classifying its contents into one of four types:

• task -- A concrete evaluation task with a task key whose value is a string name.
• group -- A collection of tasks defined by a task key whose value is a list.
• tag -- A label that can be assigned to tasks via the tag (or legacy group) field in a task config. Tags allow calling multiple tasks by theme (e.g., all perception tasks).
• python_task -- A special YAML that references a Python class via the class key, for tasks that require programmatic construction beyond what YAML can express.

This classification produces a task index -- a dictionary mapping task names to metadata including their type and the path to their YAML file. The task index serves as the single source of truth for what tasks are available.

External directories can be added to the search path via the include_path parameter, allowing users to define custom tasks outside the core repository. The include_defaults flag controls whether the built-in tasks directory is also scanned.

Usage

Use this directory structure whenever you create a new evaluation task. Create a subdirectory under lmms_eval/tasks/ with your task's name, place a YAML configuration file inside it, and optionally add a utils.py module for custom functions. The framework will automatically discover your task on the next run. For tasks maintained outside the main repository, use --include_path /path/to/custom/tasks on the command line.

Theoretical Basis

Task discovery can be modeled as a recursive directory traversal with classification:

TaskIndex = {}
for each yaml_file in walk(task_dirs):
    config = parse(yaml_file)
    if config has "class":
        TaskIndex[config["task"]] = {type: "python_task", path: yaml_file}
    elif config["task"] is list:
        TaskIndex[config["group"]] = {type: "group", path: yaml_file}
    elif config["task"] is str:
        TaskIndex[config["task"]] = {type: "task", path: yaml_file}
        for tag in config.get("tag", []):
            TaskIndex[tag].tasks.append(config["task"])

The classification is mutually exclusive at the config level: a single YAML file produces exactly one primary entry in the index (though it may also register tags as side effects). The resulting index supports O(1) lookup by name, which is critical during evaluation when many tasks may be requested simultaneously.

The design follows the Convention over Configuration principle: by placing files in the expected location with the expected structure, no additional registration step is needed.

Related Pages

Implemented By

• Implementation:EvolvingLMMs_Lab_Lmms_eval_TaskManager_Initialize_Tasks