Implementation:EvolvingLMMs Lab Lmms eval Gather Object

Knowledge Sources	lmms-eval
Domains	Distributed_Computing, Data_Processing
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for collecting per-rank evaluation results to rank 0 using collective gather operations provided by the lmms-eval framework.

Description

The result gathering logic in lmms_eval/evaluator.py (L714-756) uses torch.distributed.gather_object() to collect three types of per-rank data onto rank 0. This function serializes arbitrary Python objects (lists, dicts, etc.) via pickle and transfers them through the distributed backend, making it suitable for gathering heterogeneous evaluation results that cannot be represented as tensors.

The gathering proceeds in three stages per task:

Stage 1: Logged samples (L717-732) -- Each rank packages its logged samples (containing model outputs, ground truth, and metadata) into a list. Rank 0 prepares a receive buffer of [None] * WORLD_SIZE, while other ranks pass None as the receive buffer. After the gather, rank 0 flattens the list of lists using itertools.chain.from_iterable.

Stage 2: Per-metric scores (L734-743) -- For each (metric, filter_key) pair in the task's sample_metrics, the per-rank score list is gathered to rank 0 and flattened. This reconstructs the complete list of scores across all documents.

Stage 3: Per-sample stability metrics (L745-754) -- For tasks using k-samples mode, per-sample stability metrics (expected accuracy, consensus accuracy, internal variance, consistency rate) are gathered in the same manner.

After all three stages complete for all tasks, a dist.barrier() at L756 ensures synchronization before rank 0 proceeds to metric aggregation.

Usage

Result gathering is invoked automatically when WORLD_SIZE > 1. The condition is checked once, and the entire gathering block executes for all eval tasks. Users do not interact with this logic directly.

Code Reference

Source Location

Repository: lmms-eval
File: lmms_eval/evaluator.py
Lines: L714-754 (three gather calls), L756 (post-gather barrier)

Signature

# Stage 1: Gather logged samples
torch.distributed.gather_object(
    obj=per_rank_samples,          # list of sample dicts on this rank
    object_gather_list=full_samples,  # [None]*WORLD_SIZE on rank 0, None on others
    dst=0,
)

# Stage 2: Gather per-metric scores
torch.distributed.gather_object(
    obj=task_output.sample_metrics[metrics],  # list of scores on this rank
    object_gather_list=metric_list,           # [None]*WORLD_SIZE on rank 0, None on others
    dst=0,
)

# Stage 3: Gather per-sample stability metrics
torch.distributed.gather_object(
    obj=task_output.per_sample_metrics[metrics],
    object_gather_list=metric_list,
    dst=0,
)

Import

import torch.distributed
import itertools

I/O Contract

Inputs

Name	Type	Required	Description
obj	`list`	Yes	The per-rank data to gather: logged samples, metric scores, or stability metrics from the current rank
object_gather_list	`list` or `None`	Yes	On rank 0: a list of `[None] * WORLD_SIZE` to receive data from all ranks; on other ranks: `None`
dst	`int`	Yes	The destination rank that receives all data; always 0 in lmms-eval
RANK	`int`	Yes	Global rank of the current process
WORLD_SIZE	`int`	Yes	Total number of processes in the distributed group

Outputs

Name	Type	Description
task_output.logged_samples (rank 0)	`list`	Flattened list of all logged samples from all ranks, produced by `list(itertools.chain.from_iterable(full_samples))`
task_output.sample_metrics[key] (rank 0)	`list`	Flattened list of metric scores from all ranks for each (metric, filter_key) pair
task_output.per_sample_metrics[key] (rank 0)	`list`	Flattened list of stability metrics from all ranks for each (metric, filter_key) pair
(other ranks)	unchanged	Non-root ranks do not receive gathered data; their local data remains unchanged

Usage Examples

Basic Example

import torch.distributed as dist
import itertools

RANK = dist.get_rank()
WORLD_SIZE = dist.get_world_size()

# Example: gathering logged samples from 4 ranks
per_rank_samples = [
    {"doc_id": i, "output": f"answer_{i}"}
    for i in range(RANK, 100, WORLD_SIZE)
]

if RANK == 0:
    full_samples = [None] * WORLD_SIZE
else:
    full_samples = None

dist.gather_object(
    obj=per_rank_samples,
    object_gather_list=full_samples,
    dst=0,
)

if RANK == 0:
    # Flatten: [[rank0_samples], [rank1_samples], ...] -> [all_samples]
    all_samples = list(itertools.chain.from_iterable(full_samples))
    # all_samples contains 100 entries covering all doc_ids

Multiple Metric Gathering

# Gathering scores for multiple metrics
for (metric, filter_key), scores in task_output.sample_metrics.items():
    metric_list = [None] * WORLD_SIZE if RANK == 0 else None

    dist.gather_object(
        obj=scores,
        object_gather_list=metric_list,
        dst=0,
    )

    if RANK == 0:
        task_output.sample_metrics[(metric, filter_key)] = list(
            itertools.chain.from_iterable(metric_list)
        )

# Barrier after all gathers
dist.barrier()

Related Pages

Implements Principle

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment