Implementation:EvolvingLMMs Lab Lmms eval Accelerator Init

Knowledge Sources	lmms-eval
Domains	Distributed_Computing, Infrastructure
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for initializing a distributed process group for multi-GPU evaluation provided by the lmms-eval framework.

Description

The lmms-eval framework supports two distributed backends for multi-GPU evaluation: accelerate and torchrun. The initialization logic in __main__.py first checks whether a distributed context is already active (as happens with torchrun, which initializes the process group before the script runs). If a process group already exists, it extracts the rank directly via torch.distributed.get_rank(). Otherwise, it creates a Hugging Face Accelerator object with a custom InitProcessGroupKwargs that sets a 60000-second timeout.

The evaluator.py module reads rank and world size from environment variables (LOCAL_RANK, RANK, WORLD_SIZE) with sensible defaults (0, 0, 1 respectively) to remain functional in single-GPU mode. The api/model.py base class provides a get_rank_and_world_size() method that queries torch.distributed if initialized, falling back to instance attributes.

Usage

Use this initialization when launching multi-GPU evaluation runs via either:

# Accelerate backend
accelerate launch --num_processes=N -m lmms_eval --model ... --tasks ...

# Torchrun backend
torchrun --nproc_per_node=N -m lmms_eval --model ... --tasks ...

The choice of backend is specified via the --distributed_executor_backend argument (defaulting to "accelerate").

Code Reference

Source Location

Repository: lmms-eval
File: lmms_eval/__main__.py
Lines: L493-503

Additional related code:

File: lmms_eval/evaluator.py, Lines: L472-474
File: lmms_eval/api/model.py, Lines: L89-98

Signature

# In __main__.py (Accelerate path):
from accelerate import Accelerator
from accelerate.utils import InitProcessGroupKwargs
import datetime

kwargs_handler = InitProcessGroupKwargs(
    timeout=datetime.timedelta(seconds=60000)
)
accelerator = Accelerator(kwargs_handlers=[kwargs_handler])

# In evaluator.py (environment variable path):
local_rank = int(os.environ.get("LOCAL_RANK", 0))
global_rank = int(os.environ.get("RANK", 0))
world_size = int(os.environ.get("WORLD_SIZE", 1))

# In api/model.py (rank query method):
def get_rank_and_world_size(self) -> Tuple[int, int]:
    if dist.is_initialized():
        return dist.get_rank(), dist.get_world_size()
    return self.rank, self.world_size

Import

from accelerate import Accelerator
from accelerate.utils import InitProcessGroupKwargs
import torch.distributed as dist

I/O Contract

Inputs

Name	Type	Required	Description
timeout	`datetime.timedelta`	No (default: 60000s)	Maximum time to wait for distributed operations before raising a timeout error
distributed_executor_backend	`str`	No (default: "accelerate")	Backend to use: `"accelerate"` or `"torchrun"`
LOCAL_RANK (env var)	`int`	No (default: 0)	Local rank of the process within its node, set by the launcher
RANK (env var)	`int`	No (default: 0)	Global rank of the process across all nodes, set by the launcher
WORLD_SIZE (env var)	`int`	No (default: 1)	Total number of processes participating in the distributed run

Outputs

Name	Type	Description
accelerator	`Accelerator` or `None`	Hugging Face Accelerator object if using the accelerate backend; `None` if the process group was already initialized (torchrun)
is_main_process	`bool`	`True` if this process is rank 0 (responsible for logging, saving results, etc.)
global_rank	`int`	The global rank of the current process (0 to world_size-1)
world_size	`int`	The total number of processes in the distributed group

Usage Examples

Basic Example

import datetime
import torch
from accelerate import Accelerator
from accelerate.utils import InitProcessGroupKwargs

# Check if torchrun already initialized the process group
if torch.distributed.is_available() and torch.distributed.is_initialized():
    accelerator = None
    is_main_process = torch.distributed.get_rank() == 0
else:
    # Initialize via Accelerate with a generous timeout
    kwargs_handler = InitProcessGroupKwargs(
        timeout=datetime.timedelta(seconds=60000)
    )
    accelerator = Accelerator(kwargs_handlers=[kwargs_handler])
    is_main_process = accelerator.is_main_process

# In the evaluator, rank info comes from environment variables
import os
local_rank = int(os.environ.get("LOCAL_RANK", 0))
global_rank = int(os.environ.get("RANK", 0))
world_size = int(os.environ.get("WORLD_SIZE", 1))

Related Pages

Implements Principle

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment