Principle:EvolvingLMMs Lab Lmms eval Parallel Inference
| Knowledge Sources | |
|---|---|
| Domains | Distributed_Computing, Model_Inference |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Executing model inference simultaneously across all distributed ranks, with explicit synchronization barriers, enables linear scaling of evaluation throughput with the number of GPUs.
Description
Once data has been sharded and requests have been padded to equal lengths, the actual model inference proceeds in parallel across all ranks. Each rank independently executes the same inference logic on its own subset of requests, and synchronization barriers ensure that all ranks complete before moving to the next phase.
The parallel inference phase involves three key concepts:
1. Request Type Dispatch
Evaluation tasks produce different types of model requests depending on the task's output type:
- generate_until -- Open-ended generation tasks where the model produces text until a stop condition is met (e.g., VQA, captioning).
- loglikelihood -- Scoring tasks where the model computes the log probability of a given continuation (e.g., multiple-choice questions).
- generate_until_multi_round -- Multi-turn dialogue tasks requiring sequential generation and conditioning.
Each request type is handled by a separate method on the model object, and all requests of the same type are batched together for efficient processing.
2. Synchronous Execution
Although each rank processes its own data independently, the distributed framework (FSDP/DDP) requires all ranks to participate in the same number of forward passes. This is why request padding (see Request_Padding) is essential. Within each forward pass, the framework may perform inter-rank communication (e.g., gradient synchronization in DDP, parameter gathering in FSDP), so all ranks must be executing forward passes simultaneously.
3. Post-Inference Barriers
After all requests of a given type have been processed, a synchronization barrier ensures all ranks have finished before proceeding to the next request type or to result collection. A second barrier after any post-inference setup (such as launching an evaluation server) ensures all ranks are ready for the result gathering phase.
Usage
Parallel inference is the core execution phase of multi-GPU evaluation. It occurs after:
- Environment initialization
- Data sharding
- Request padding
And before:
- Result gathering
- Metric aggregation
The degree of parallelism is determined entirely by the number of processes launched (world_size). No code changes are needed to scale from 1 GPU to N GPUs.
Theoretical Basis
The theoretical speedup from parallel inference follows Amdahl's Law, modified for the evaluation context:
Let:
T_1 = time for single-GPU evaluation
T_infer = time spent on model inference (parallelizable)
T_other = time for data loading, metric computation, etc. (serial)
W = world_size (number of GPUs)
T_1 = T_infer + T_other
Ideal parallel time:
T_W = T_infer / W + T_other
Speedup:
S(W) = T_1 / T_W = (T_infer + T_other) / (T_infer / W + T_other)
Since model inference typically dominates evaluation time (T_infer >> T_other), the speedup approaches W for large models and datasets.
The synchronization cost at each barrier is bounded by the difference between the fastest and slowest rank's inference time:
T_barrier <= max(T_infer_r) - min(T_infer_r) for r in {0, ..., W-1}
Request padding minimizes this difference by ensuring equal request counts, though per-request inference time may still vary due to different input sizes.