Implementation:Mlfoundations Open flamingo Get data
Overview
Concrete tool for constructing WebDataset-based data loading pipelines for LAION and MMC4 datasets provided by the OpenFlamingo training module.
Description
The get_data() function dispatches to get_laion_dataset() or get_mmc4_dataset() based on dataset_type. LAION pipeline: decode images as PIL RGB → random horizontal flip → tokenize captions (max 32 tokens) → batch. MMC4 pipeline: decode base64 images → image-text similarity filtering via Hungarian matching → tokenize interleaved text (max 256 tokens) → batch. Both use DataInfo wrapper containing the WebLoader dataloader and a SharedEpoch for deterministic shuffling.
Usage
Called in the training script to create dataloaders before the training loop.
Code Reference
Source: Repository https://github.com/mlfoundations/open_flamingo, File: open_flamingo/train/data.py Lines L486-492 (get_data), L372-471 (get_laion_dataset), L271-369 (get_mmc4_dataset)
Signature:
def get_data(args, image_processor, tokenizer, dataset_type: str, epoch: int = 0) -> DataInfo:
"""
Interface for getting the webdatasets.
Args:
args: training args with shard paths, batch sizes, etc.
image_processor: CLIP image preprocessing function
tokenizer: tokenizer with special tokens
dataset_type: "image_text" (LAION) or "mmc4"
epoch: current epoch for deterministic shuffling
Returns:
DataInfo with .dataloader (WebLoader) and .shared_epoch (SharedEpoch)
"""
Import:
from open_flamingo.train.data import get_data
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| args | argparse.Namespace |
Yes | Training args with laion_shards/mmc4_shards paths, batch_size_laion/batch_size_mmc4, etc.
|
| image_processor | Callable |
Yes | CLIP image preprocessor |
| tokenizer | PreTrainedTokenizer |
Yes | Tokenizer with special tokens |
| dataset_type | str |
Yes | "image_text" or "mmc4"
|
| epoch | int |
No | Current epoch (default 0) |
Outputs
| Name | Type | Description |
|---|---|---|
| DataInfo | DataInfo |
Contains .dataloader (WebLoader yielding batches of (images_tensor, (input_ids, attention_mask))) and .shared_epoch for epoch synchronization
|
Usage Examples
Creating both LAION and MMC4 dataloaders:
from open_flamingo.train.data import get_data
# Create LAION dataloader (single image-text pairs)
laion_data = get_data(
args=args,
image_processor=image_processor,
tokenizer=tokenizer,
dataset_type="image_text",
epoch=0,
)
laion_loader = laion_data.dataloader
# Create MMC4 dataloader (interleaved multi-image documents)
mmc4_data = get_data(
args=args,
image_processor=image_processor,
tokenizer=tokenizer,
dataset_type="mmc4",
epoch=0,
)
mmc4_loader = mmc4_data.dataloader
# Training loop
for epoch in range(num_epochs):
laion_data.shared_epoch.set_epoch(epoch)
mmc4_data.shared_epoch.set_epoch(epoch)
for laion_batch, mmc4_batch in zip(laion_loader, mmc4_loader):
images, (input_ids, attention_mask) = laion_batch
# ... training step ...