Implementation:EvolvingLMMs Lab Lmms eval TaskManager Initialize Tasks

Knowledge Sources	lmms-eval
Domains	Task_Management, Configuration
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for discovering and indexing evaluation tasks from directory structures provided by the lmms-eval framework.

Description

The TaskManager.initialize_tasks() method builds a comprehensive index of all available evaluation tasks by scanning one or more directories for YAML configuration files. It delegates the actual directory traversal and file classification to the internal _get_task_and_group() method.

The TaskManager class serves as the central registry for all tasks. During initialization, it calls initialize_tasks() to populate its internal index, then categorizes entries into groups, subtasks, and tags for quick lookup. The resulting index maps each task name to a metadata dictionary containing the task's type ("task", "group", "tag", or "python_task") and the path to its YAML configuration file.

The _get_task_and_group() method performs the recursive directory walk, skipping __pycache__ and .ipynb_checkpoints directories. For each YAML file found, it loads the configuration in "simple" mode (without resolving includes or !function directives) and classifies it based on the presence and type of the "task", "group", and "class" keys.

Usage

Use this when you need to discover available tasks programmatically, add external task directories, or build tooling that lists or inspects registered tasks. The TaskManager is instantiated automatically during evaluation, but you can also create one manually for inspection.

Code Reference

Source Location

• Repository: lmms-eval
• File: lmms_eval/tasks/__init__.py
• Lines: 46-75 (initialize_tasks), 384-486 (_get_task_and_group)

Signature

def initialize_tasks(
    self,
    include_path: Optional[Union[str, List]] = None,
    include_defaults: bool = True,
) -> dict:

Import

from lmms_eval.tasks import TaskManager

I/O Contract

Inputs

Name	Type	Required	Description
include_path	Optional[Union[str, List]]	No	One or more additional directory paths to scan for task YAML files. These are searched in addition to (or instead of) the default lmms_eval/tasks/ directory.
include_defaults	bool	No	If True (default), the built-in lmms_eval/tasks/ directory is included in the search. Set to False to only index tasks from include_path.

Outputs

Name	Type	Description
task_index	dict	A dictionary mapping task names (strings) to metadata dictionaries. Each metadata dict contains: "type" (one of "task", "group", "tag", "python_task"), "yaml_path" (path to the YAML file, or -1 for tags and groups defined inline), and optionally "task" (a list of subtask names for groups and tags).

Usage Examples

Basic Example

from lmms_eval.tasks import TaskManager

# Initialize with defaults only
tm = TaskManager()
print(tm.all_tasks)        # All registered task/group/tag names
print(tm.all_subtasks)     # Only concrete tasks
print(tm.all_groups)       # Only groups

# Check a specific task
if "mme" in tm.task_index:
    info = tm.task_index["mme"]
    print(info["type"])       # "task"
    print(info["yaml_path"])  # path to mme.yaml

With External Task Directory

from lmms_eval.tasks import TaskManager

# Include a custom tasks directory alongside defaults
tm = TaskManager(
    include_path="/path/to/my/custom_tasks",
    include_defaults=True,
)
# Custom tasks are now indexed alongside built-in tasks
print(tm.all_tasks)

Listing All Tasks

from lmms_eval.tasks import TaskManager

tm = TaskManager()
# Produce a formatted table of all tasks, groups, and tags
table = tm.list_all_tasks(
    list_groups=True,
    list_tags=True,
    list_subtasks=True,
)
print(table)

Related Pages

Implements Principle

• Principle:EvolvingLMMs_Lab_Lmms_eval_Task_Directory_Structure