Implementation:OpenGVLab InternVL Train Sampler Patch

Knowledge Sources	OpenGVLab_InternVL
Domains	Data Sampling, Training, Distributed Training
Last Updated	2026-02-07 14:00 GMT

Overview

Patches the HuggingFace Trainer with a custom LengthGroupedSampler that groups training samples by sequence length for more efficient batching in multimodal training.

Description

This module implements length-grouped sampling adapted from the LLaVA project, consisting of:

split_to_even_chunks divides a list of indices into num_chunks chunks of roughly equal total length. It distributes indices by always assigning the next index to the shortest chunk until each chunk reaches its capacity.

get_length_grouped_indices creates mega-batches of size world_size * batch_size, sorts each mega-batch by sequence length in descending order, then splits into world-size-even chunks. This ensures that each GPU receives samples of similar length within a batch.

LengthGroupedSampler class extends torch.utils.data.Sampler:

• Accepts batch_size, world_size, and either a dataset or pre-computed lengths.
• If lengths are not provided, infers them from the dataset's model_input_name key.
• The __iter__ method yields indices via get_length_grouped_indices.

The patched _get_train_sampler method:

• When group_by_length is enabled, aggregates lengths from all sub-datasets (supporting concatenated datasets) and creates a LengthGroupedSampler with world_size * gradient_accumulation_steps as the effective world size.
• Otherwise falls back to RandomSampler.

replace_train_sampler monkey-patches this method onto transformers.Trainer.

Usage

Call replace_train_sampler() before creating a HuggingFace Trainer to enable length-grouped sampling. This is especially important for multimodal training where sequence lengths vary significantly due to different numbers of image tiles per sample.

Code Reference

Source Location

• Repository: OpenGVLab_InternVL
• File: internvl_chat/internvl/patch/train_sampler_patch.py
• Lines: 1-125

Signature

def split_to_even_chunks(indices, lengths, num_chunks) -> list: ...

def get_length_grouped_indices(lengths, batch_size, world_size,
                                generator=None, merge=True) -> list: ...

class LengthGroupedSampler(Sampler):
    def __init__(self, batch_size: int, world_size: int,
                 dataset=None, lengths=None, model_input_name=None,
                 generator=None): ...
    def __len__(self) -> int: ...
    def __iter__(self): ...

def _get_train_sampler(self) -> Optional[Sampler]: ...

def replace_train_sampler(): ...

Import

from internvl.patch.train_sampler_patch import replace_train_sampler

I/O Contract

Inputs

Name	Type	Required	Description
batch_size	int	Yes	Batch size per device
world_size	int	Yes	Total number of distributed workers (times gradient accumulation steps)
lengths	List[int]	No	Pre-computed sequence lengths; inferred from dataset if not provided
dataset	Dataset	No	Training dataset (required if lengths not provided)

Outputs

Name	Type	Description
sampler	LengthGroupedSampler	A sampler that yields indices grouped by sequence length

Usage Examples

Basic Usage

from internvl.patch.train_sampler_patch import replace_train_sampler

# Patch the HuggingFace Trainer before creating it
replace_train_sampler()

# Now create the Trainer normally - it will use length-grouped sampling
trainer = Trainer(
    model=model,
    args=training_args,  # with group_by_length=True
    train_dataset=train_dataset,
)

Related Pages

• Principle:OpenGVLab_InternVL_Length_Grouped_Sampling