Implementation:Huggingface Datasets DatasetBuilder Download and Prepare
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, NLP |
| Last Updated | 2026-02-14 18:00 GMT |
Overview
Concrete tool for downloading raw data and converting it to an efficient on-disk format (Arrow/Parquet) provided by the HuggingFace Datasets library.
Description
DatasetBuilder.download_and_prepare is the primary method that drives the full data acquisition and serialization pipeline for any dataset. When called, it creates or reuses a DownloadManager to fetch remote files, invokes the dataset-specific _split_generators and _generate_examples methods to produce structured records, writes those records to sharded Arrow or Parquet files in a temporary directory, records metadata (checksums, split sizes, feature schemas), and atomically renames the temporary directory to the final cache location. It supports local and remote (S3, GCS) output directories, configurable shard sizes, multi-process downloading, and multiple download/verification modes.
Usage
Call download_and_prepare() when you need to materialize a dataset on disk before constructing in-memory Dataset objects via as_dataset(). This is typically done automatically by load_dataset(), but can be invoked explicitly when using load_dataset_builder() for fine-grained control over output location, format, or shard size.
Code Reference
Source Location
- Repository: datasets
- File:
src/datasets/builder.py - Lines: L683-L904
Signature
def download_and_prepare(
self,
output_dir: Optional[str] = None,
download_config: Optional[DownloadConfig] = None,
download_mode: Optional[Union[DownloadMode, str]] = None,
verification_mode: Optional[Union[VerificationMode, str]] = None,
dl_manager: Optional[DownloadManager] = None,
base_path: Optional[str] = None,
file_format: str = "arrow",
max_shard_size: Optional[Union[int, str]] = None,
num_proc: Optional[int] = None,
storage_options: Optional[dict] = None,
**download_and_prepare_kwargs,
):
Import
from datasets import load_dataset_builder
# Access via builder instance:
builder = load_dataset_builder("dataset_name")
builder.download_and_prepare()
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| output_dir | str |
No | Output directory for the dataset. Defaults to the builder's cache_dir inside ~/.cache/huggingface/datasets.
|
| download_config | DownloadConfig |
No | Specific download configuration parameters (cache dir, force download, etc.). |
| download_mode | DownloadMode or str |
No | Select the download/generate mode. Defaults to REUSE_DATASET_IF_EXISTS.
|
| verification_mode | VerificationMode or str |
No | Determines the checks to run on downloaded/processed dataset info (checksums/size/splits). Defaults to BASIC_CHECKS.
|
| dl_manager | DownloadManager |
No | Specific DownloadManager instance to use.
|
| base_path | str |
No | Base path for relative paths used to download files. Can be a remote URL. |
| file_format | str |
No | Format of the output data files: "arrow" (default) or "parquet".
|
| max_shard_size | Union[str, int] |
No | Maximum bytes per shard file. Default is "500MB". Based on uncompressed data size.
|
| num_proc | int |
No | Number of processes for downloading and generating the dataset. Defaults to None (single process).
|
| storage_options | dict |
No | Key/value pairs passed to the caching file-system backend (e.g., S3 credentials). |
| **download_and_prepare_kwargs | keyword args | No | Additional keyword arguments forwarded to internal methods. |
Outputs
| Name | Type | Description |
|---|---|---|
| (return value) | None |
The method does not return a value. Its side effect is writing dataset files (Arrow or Parquet shards plus metadata) to output_dir.
|
Usage Examples
Basic Usage
from datasets import load_dataset_builder
builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
builder.download_and_prepare()
ds = builder.as_dataset(split="train")
Parquet Output to Custom Directory
from datasets import load_dataset_builder
builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
builder.download_and_prepare("./output_dir", file_format="parquet")
Remote Storage (S3)
from datasets import load_dataset_builder
storage_options = {"key": "aws_access_key_id", "secret": "aws_secret_access_key"}
builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes")
builder.download_and_prepare(
"s3://my-bucket/my_rotten_tomatoes",
storage_options=storage_options,
file_format="parquet",
)