Implementation:EvolvingLMMs Lab Lmms eval Padding Requests
| Knowledge Sources | |
|---|---|
| Domains | Distributed_Computing, Evaluation |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Concrete tool for equalizing request counts across ranks by padding shorter lists to match the longest provided by the lmms-eval framework.
Description
The padding logic in lmms_eval/evaluator.py operates in two phases:
Phase 1: Count gathering (L547-564) -- After each task's requests are built, the evaluator counts the number of instances on the local rank and uses a collective operation to gather these counts from all ranks. For the accelerate backend, lm.accelerator.gather(tensor) is used. For the torchrun backend, torch.distributed.all_gather_into_tensor() is used. The result is a list of instance counts indexed by rank.
Phase 2: Padding application (L575-577) -- When dispatching requests for inference, if the current rank needs padding, the last request in the list is duplicated the required number of times. This is done by appending copies of the last cloned request to the cloned_reqs list. The padded requests go through the same inference pipeline as real requests, and their results are stored in the resps list. However, since these padded requests are appended after the real requests, they are implicitly ignored during result collection -- only the first n_r responses correspond to actual evaluation instances.
A special mapping handles multiple-choice tasks: when task.OUTPUT_TYPE == "multiple_choice", the request type is mapped to "loglikelihood" because multiple-choice tasks dispatch multiple loglikelihood requests per document.
Usage
Padding is applied automatically whenever world_size > 1. No user configuration is required. The padding dictionary (padding_requests) accumulates padding counts across all tasks, keyed by request type (e.g., "generate_until", "loglikelihood").
Code Reference
Source Location
- Repository: lmms-eval
- File:
lmms_eval/evaluator.py - Lines: L463 (padding_requests dict), L547-564 (padding calculation), L575-577 (padding application)
Signature
# Phase 1: Gathering instance counts (accelerate backend)
instances_rnk = torch.tensor(len(task._instances), device=lm.device)
gathered_item = lm.accelerator.gather(instances_rnk).cpu().detach().numpy().tolist()
# Phase 1: Gathering instance counts (torchrun backend)
instances_rnk = torch.tensor(len(task._instances), device=lm.device)
gathered_item = torch.zeros(world_size * 1, dtype=instances_rnk.dtype, device=lm.device)
dist.all_gather_into_tensor(gathered_item, instances_rnk)
gathered_item = gathered_item.cpu().detach().numpy().tolist()
# Compute padding amount
reqtype = "loglikelihood" if task.OUTPUT_TYPE == "multiple_choice" else task.OUTPUT_TYPE
numpad = max(gathered_item) - gathered_item[lm.rank]
padding_requests[reqtype] += numpad
# Phase 2: Apply padding during dispatch
if (world_size > 1) and (padding_requests[reqtype] > 0):
for _ in range(padding_requests[reqtype]):
cloned_reqs.extend([req] * req.repeats)
Import
import torch
import torch.distributed as dist
from collections import defaultdict
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| task._instances | list |
Yes | List of Instance objects for the current task on the local rank |
| lm.device | torch.device |
Yes | The device of the model, used for tensor creation |
| lm.rank | int |
Yes | The rank index of the current process |
| world_size | int |
Yes | Total number of distributed processes |
| distributed_executor_backend | str |
Yes | Either "accelerate" or "torchrun"
|
| task.OUTPUT_TYPE | str |
Yes | The output type of the task (e.g., "generate_until", "multiple_choice")
|
Outputs
| Name | Type | Description |
|---|---|---|
| padding_requests | defaultdict(int) |
Dictionary mapping request types to the number of padding instances needed on this rank |
| cloned_reqs (padded) | list |
The request list extended with duplicates of the last request to equalize across ranks |
Usage Examples
Basic Example
import torch
from collections import defaultdict
# Suppose world_size=4 and ranks have instance counts [50, 52, 51, 50]
# On rank 0:
padding_requests = defaultdict(int)
instances_rnk = torch.tensor(50, device="cuda:0")
# After all_gather, gathered_item = [50, 52, 51, 50]
gathered_item = [50, 52, 51, 50]
reqtype = "generate_until"
numpad = max(gathered_item) - gathered_item[0] # 52 - 50 = 2
padding_requests[reqtype] += numpad # padding_requests["generate_until"] = 2
# Later, during dispatch, 2 copies of the last request are appended
cloned_reqs = [...] # 50 real requests
if padding_requests[reqtype] > 0:
for _ in range(padding_requests[reqtype]):
cloned_reqs.append(cloned_reqs[-1])
# cloned_reqs now has 52 entries, matching the max across all ranks