Principle:OpenGVLab InternVL Length Grouped Sampling
| Knowledge Sources | |
|---|---|
| Domains | Data Sampling, Training Efficiency, Distributed Training |
| Last Updated | 2026-02-07 14:00 GMT |
Overview
Length Grouped Sampling groups training samples by sequence length within each batch to minimize padding waste, improving GPU utilization and training throughput for variable-length multimodal data.
Description
In multimodal training, sequence lengths vary significantly because different samples may contain different numbers of image tiles, text tokens, or both. When samples of very different lengths are batched together, shorter sequences must be padded to match the longest sequence in the batch, wasting computation on padding tokens.
Length Grouped Sampling addresses this by:
- Mega-batch creation -- Random permutation of the full dataset is partitioned into mega-batches of size world_size * batch_size.
- Length-based sorting -- Each mega-batch is sorted by sequence length in descending order, grouping similarly-sized samples together.
- Even chunk splitting -- Sorted mega-batches are split into world_size chunks balanced by total length, ensuring each GPU receives a fair share of the computational load.
- Distributed awareness -- The effective world size accounts for gradient accumulation steps, so that all micro-batches within an accumulation window contain similarly-sized samples.
This approach maintains a degree of randomness (samples are randomly assigned to mega-batches) while significantly reducing within-batch length variance and thus padding overhead.
Usage
Apply length-grouped sampling in multimodal training pipelines where sequence lengths vary significantly. Enable it by setting group_by_length=True in the HuggingFace Trainer arguments and patching the Trainer's sampler before training begins.
Theoretical Basis
Length-grouped sampling is a form of curriculum-aware batching that optimizes hardware utilization without biasing the training distribution. By grouping similarly-sized samples, the approach minimizes the ratio of padding tokens to content tokens per batch, directly reducing wasted FLOPs. The mega-batch randomization preserves the stochastic nature of SGD, ensuring that the model sees diverse data within each epoch while benefiting from efficient batching. This technique was popularized by the LLaVA project and is widely adopted in multimodal model training.