Principle:Huggingface Datasets Dataset Interleaving
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, NLP |
| Last Updated | 2026-02-14 18:00 GMT |
Overview
Dataset Interleaving is the principle of alternating between multiple source datasets to produce a single interleaved dataset, with optional sampling probabilities and configurable stopping strategies.
Description
When training on multiple data sources simultaneously, it is often desirable to interleave examples from each source rather than concatenating them sequentially. The Dataset Interleaving principle covers round-robin interleaving (cycling through sources in order) and probabilistic sampling (drawing from sources according to specified probabilities with a seed for reproducibility). Three stopping strategies control when the interleaved dataset ends: "first_exhausted" (stop when any source runs out), "all_exhausted" (oversample until all sources have been fully consumed at least once), and "all_exhausted_without_replacement" (each sample appears at most once). The function works with both map-style and iterable datasets.
Usage
Use Dataset Interleaving when training on a mixture of datasets and you want to control the sampling ratio between sources. This is common in multilingual training (balancing high-resource and low-resource languages), multi-task learning, and domain adaptation where the mixing ratio affects model performance.
Theoretical Basis
Interleaving implements a multiplexing strategy over multiple data streams. With uniform probabilities (or no probabilities), interleaving reduces to round-robin scheduling, which ensures equal representation of each source. With non-uniform probabilities, interleaving implements a weighted random sampling scheme where the probability of drawing from source i at each step is p_i. The stopping strategy determines the bias-variance tradeoff: "first_exhausted" avoids oversampling but may underrepresent large datasets, while "all_exhausted" ensures full coverage at the cost of duplicating examples from smaller sources.