Heuristic:EvolvingLMMs Lab Lmms eval Distributed Padding Strategy
| Knowledge Sources | |
|---|---|
| Domains | Distributed_Training, Optimization |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Padding requests with dummy examples to ensure equal workload across distributed ranks, preventing FSDP/DDP synchronization hangs.
Description
In distributed evaluation (multi-GPU), different ranks may receive unequal numbers of evaluation instances due to dataset partitioning. Since DDP and FSDP require all ranks to participate in collective operations (barriers, all-gather), a rank that finishes early will hang waiting for others. The lmms-eval framework solves this by computing the maximum instance count across all ranks, then padding shorter ranks with dummy requests to equalize the workload. After inference, only the non-padded results are retained.
Usage
This heuristic is automatically applied when world_size > 1 (distributed evaluation). It is critical for the Distributed Multi-GPU Evaluation workflow. Users do not need to configure it manually — the framework handles padding transparently.
The Insight (Rule of Thumb)
- Action: Gather instance counts across all ranks, compute the maximum, and pad each rank's request list to match the maximum.
- Value:
numpad = max(gathered_item) - gathered_item[lm.rank] - Trade-off: Slight overhead from processing dummy requests, but prevents indefinite hanging in distributed mode.
- Caveat: The current implementation may not handle tasks with multiple request types (e.g., SquadV2-like tasks) — this is noted as a TODO in the source code.
Reasoning
DDP and FSDP require all ranks to execute the same number of forward passes so that gradient synchronization barriers can be met. If rank 0 has 100 instances and rank 1 has 98, rank 0 will finish 2 passes after rank 1 and call a barrier that rank 1 never reaches, causing a deadlock. Padding eliminates this asymmetry. The dummy results are discarded after the all-gather step, so they do not affect evaluation metrics.
The framework supports two synchronization backends:
- accelerate: Uses
lm.accelerator.gather()to exchange instance counts - torchrun: Uses
dist.all_gather_into_tensor()for the same purpose
Code evidence from lmms_eval/evaluator.py:547-577:
if world_size > 1:
if distributed_executor_backend == "accelerate":
instances_rnk = torch.tensor(len(task._instances), device=lm.device)
gathered_item = lm.accelerator.gather(instances_rnk).cpu().detach().numpy().tolist()
elif distributed_executor_backend == "torchrun":
instances_rnk = torch.tensor(len(task._instances), device=lm.device)
gathered_item = torch.zeros(world_size * 1, dtype=instances_rnk.dtype, device=lm.device)
dist.all_gather_into_tensor(gathered_item, instances_rnk)
# "multiple_choice" task types dispatch (several) "loglikelihood" request types
reqtype = "loglikelihood" if task.OUTPUT_TYPE == "multiple_choice" else task.OUTPUT_TYPE
# compute number of pseudo-batches to pad with (FSDP/DDP require even batches among ranks)
numpad = max(gathered_item) - gathered_item[lm.rank]
# todo: may not account for padding in cases like SquadV2 which has multiple req types
padding_requests[reqtype] += numpad
Padding application from lmms_eval/evaluator.py:575-577:
if (world_size > 1) and (padding_requests[reqtype] > 0):
for _ in range(padding_requests[reqtype]):
cloned_reqs.extend([req] * req.repeats)