Implementation:EvolvingLMMs Lab Lmms eval TaskManager Match Tasks

Knowledge Sources	lmms-eval
Domains	Evaluation, Task_Management
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for resolving task name patterns to concrete evaluation task configurations, provided by the lmms-eval framework.

Description

The TaskManager class is the central registry for all evaluation tasks in lmms-eval. On initialization, it recursively walks the default task directory (lmms_eval/tasks/) and any user-specified include_path directories, parsing every YAML file to build a task index. Each entry in the index records whether it is a task, python_task, group, or tag, along with the YAML file path.

The match_tasks() method delegates to utils.pattern_match(), which uses fnmatch.filter() to expand glob patterns against the list of all indexed task names. The result is a sorted, deduplicated list of matched task name strings.

After matching, load_task_or_group() instantiates the actual ConfigurableTask objects by loading and merging YAML configs, resolving group hierarchies, and applying task-type-specific overrides.

Usage

Use TaskManager.match_tasks() when you need to:

• Expand glob patterns from the CLI --tasks argument into concrete task names.
• Validate that user-supplied task names exist in the index before launching evaluation.
• Build custom task selection logic in programmatic evaluation pipelines.

Code Reference

Source Location

• Repository: lmms-eval
• File: lmms_eval/tasks/__init__.py
• Lines: 19-486 (TaskManager class), 159-160 (match_tasks method)

Signature

class TaskManager:
    def __init__(
        self,
        verbosity: str = "INFO",
        include_path: Optional[Union[str, List]] = None,
        include_defaults: bool = True,
        model_name: Optional[str] = None,
    ) -> None: ...

    def match_tasks(self, task_list: list) -> list[str]:
        return utils.pattern_match(task_list, self.all_tasks)

    def load_task_or_group(
        self,
        task_list: Optional[Union[str, list]] = None,
        task_type: Literal["simple", "chat"] = "simple",
    ) -> dict: ...

Import

from lmms_eval.tasks import TaskManager

I/O Contract

Inputs

Name	Type	Required	Description
task_list	list[str]	Yes	List of task name strings or glob patterns (e.g., ["mmmu*", "mme"])
include_path	Optional[Union[str, List]]	No	Additional directory paths to search for task YAML configs
include_defaults	bool	No	Whether to include the default lmms_eval/tasks/ directory (default: True)
model_name	Optional[str]	No	Model name used for model-specific task configuration overrides

Outputs

Name	Type	Description
matched_tasks	list[str]	Sorted list of task name strings matching the input patterns
task_dict	dict	Dictionary mapping task names to instantiated ConfigurableTask objects (from load_task_or_group)

Usage Examples

Basic Example

from lmms_eval.tasks import TaskManager

# Initialize task manager with default tasks
tm = TaskManager(verbosity="INFO")

# Match specific tasks using glob patterns
matched = tm.match_tasks(["mmmu*", "mme"])
print(matched)
# ['mme', 'mmmu_pro_standard_vision', 'mmmu_val', ...]

# List all available tasks
print(len(tm.all_tasks))

# Load matched tasks as ConfigurableTask objects
task_dict = tm.load_task_or_group(matched)

With Custom Task Directory

from lmms_eval.tasks import TaskManager

# Include custom task definitions alongside defaults
tm = TaskManager(
    include_path="/path/to/custom/tasks",
    include_defaults=True,
    model_name="qwen2_5_vl",
)

# Match tasks from both default and custom directories
matched = tm.match_tasks(["my_custom_task", "mmmu*"])
task_dict = tm.load_task_or_group(matched)

Related Pages

Implements Principle

• Principle:EvolvingLMMs_Lab_Lmms_eval_Task_Selection