Implementation:Huggingface Datasets Load Dataset Streaming

Knowledge Sources	Huggingface Datasets HF Datasets Docs
Domains	Data_Engineering, NLP
Last Updated	2026-02-14 18:00 GMT

Overview

Concrete tool for loading a dataset in streaming mode provided by the HuggingFace Datasets library.

Description

The load_dataset function with streaming=True returns an IterableDataset (or IterableDatasetDict if no split is specified) instead of a regular Dataset. When streaming is enabled, the function follows a truncated execution path:

1. It builds a dataset builder via load_dataset_builder, which identifies the data format and file layout.
2. Instead of calling builder_instance.download_and_prepare() (the standard eager path), it calls builder_instance.as_streaming_dataset(split=split).
3. The builder constructs an IterableDataset backed by a StreamingDownloadManager that resolves URLs lazily.

The streaming=True path explicitly forbids combining with num_proc (parallel downloading is not supported in streaming mode; use PyTorch DataLoader num_workers instead). Parameters like download_mode, verification_mode, and keep_in_memory are ignored or irrelevant when streaming.

Usage

Use load_dataset(..., streaming=True) whenever you want to iterate over data without downloading it to disk. This is the primary entry point for the entire streaming workflow.

Code Reference

Source Location

• Repository: datasets
• File: src/datasets/load.py
• Lines: L1278-L1519

Signature

def load_dataset(
    path: str,
    name: Optional[str] = None,
    data_dir: Optional[str] = None,
    data_files: Optional[Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]] = None,
    split: Optional[Union[str, Split, list[str], list[Split]]] = None,
    cache_dir: Optional[str] = None,
    features: Optional[Features] = None,
    download_config: Optional[DownloadConfig] = None,
    download_mode: Optional[Union[DownloadMode, str]] = None,
    verification_mode: Optional[Union[VerificationMode, str]] = None,
    keep_in_memory: Optional[bool] = None,
    save_infos: bool = False,
    revision: Optional[Union[str, Version]] = None,
    token: Optional[Union[bool, str]] = None,
    streaming: bool = False,
    num_proc: Optional[int] = None,
    storage_options: Optional[dict] = None,
    **config_kwargs,
) -> Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]:

Import

from datasets import load_dataset

I/O Contract

Inputs

Name	Type	Required	Description
path	str	Yes	Path or name of the dataset (Hub repo id, local directory, or builder name).
name	Optional[str]	No	Name of the dataset configuration.
data_dir	Optional[str]	No	Subdirectory of the dataset to load.
data_files	Optional[Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]]]]	No	Explicit path(s) to source data file(s).
split	Optional[Union[str, Split, list[str], list[Split]]]	No	Which split to load. If None, returns a dict of all splits.
cache_dir	Optional[str]	No	Directory for cached data (unused in streaming mode).
features	Optional[Features]	No	Feature types to impose on the dataset.
download_config	Optional[DownloadConfig]	No	Download configuration parameters.
download_mode	Optional[Union[DownloadMode, str]]	No	Download/generate mode.
verification_mode	Optional[Union[VerificationMode, str]]	No	Verification checks to run.
keep_in_memory	Optional[bool]	No	Whether to copy dataset in-memory (unused in streaming mode).
save_infos	bool	No	Whether to save dataset info (defaults to False).
revision	Optional[Union[str, Version]]	No	Version of the dataset to load.
token	Optional[Union[bool, str]]	No	Authentication token for the Hugging Face Hub.
streaming	bool	No	Must be set to True for streaming mode. Defaults to False.
num_proc	Optional[int]	No	Not supported with streaming; raises NotImplementedError.
storage_options	Optional[dict]	No	Key/value pairs for dataset file-system backend.

Outputs

Name	Type	Description
dataset	IterableDataset or IterableDatasetDict	A lazily-loaded streaming dataset. Returns IterableDataset when a specific split is requested, or IterableDatasetDict when split is None.

Usage Examples

Basic Usage

from datasets import load_dataset

# Stream a dataset from the Hugging Face Hub
ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True)

# Iterate over the first 3 examples
for example in ds.take(3):
    print(example)

Related Pages

Implements Principle

• Principle:Huggingface_Datasets_Streaming_Dataset_Loading