Implementation:Huggingface Datasets Cache Builder

Knowledge Sources	Huggingface Datasets HF Datasets Docs
Domains	Caching, Data_Loading
Last Updated	2026-02-14 18:00 GMT

Overview

Dataset builder that loads previously cached datasets from the local filesystem, provided by the HuggingFace Datasets library.

Description

Cache is a specialized dataset builder extending datasets.ArrowBasedBuilder that reconstructs a dataset from an existing cache directory rather than downloading and processing data from scratch. It is used internally by the load_dataset pipeline when cached data is available.

The module also provides the helper function _find_hash_in_cache, which locates the most recently modified cached dataset directory matching a given dataset name, config, and optional custom features. This function searches the cache directory hierarchy (cache_dir/dataset_name/config_id/version/hash/) and returns the resolved (config_name, version, hash) tuple.

The Cache builder's __init__ accepts either a repo_id or dataset_name (at least one required) along with optional config_name, version, and hash parameters. When both version and hash are set to "auto", it invokes _find_hash_in_cache to resolve the cache location automatically.

The download_and_prepare method is overridden to simply validate that the cache directory exists (and optionally copy it to an output directory), bypassing the normal download-and-process pipeline. The _split_generators method reads split information from the cached dataset_info.json and generates file paths for each split's Arrow shards.

Usage

The Cache builder is used internally by load_dataset when loading from cache. End users typically do not instantiate it directly, but understanding its behavior is useful for debugging cache-related issues.

Code Reference

Source Location

• Repository: datasets
• File: src/datasets/packaged_modules/cache/cache.py
• Lines: 1-195

Signature

def _find_hash_in_cache(
    dataset_name: str,
    config_name: Optional[str],
    cache_dir: Optional[str],
    config_kwargs: dict,
    custom_features: Optional[datasets.Features],
) -> tuple[str, str, str]:

class Cache(datasets.ArrowBasedBuilder):
    def __init__(
        self,
        cache_dir: Optional[str] = None,
        dataset_name: Optional[str] = None,
        config_name: Optional[str] = None,
        version: Optional[str] = "0.0.0",
        hash: Optional[str] = None,
        base_path: Optional[str] = None,
        info: Optional[datasets.DatasetInfo] = None,
        features: Optional[datasets.Features] = None,
        token: Optional[Union[bool, str]] = None,
        repo_id: Optional[str] = None,
        data_files: Optional[Union[str, list, dict, datasets.data_files.DataFilesDict]] = None,
        data_dir: Optional[str] = None,
        storage_options: Optional[dict] = None,
        writer_batch_size: Optional[int] = None,
        **config_kwargs,
    ):

Key methods:

def _info(self) -> datasets.DatasetInfo:
    return datasets.DatasetInfo()

def download_and_prepare(self, output_dir: Optional[str] = None, *args, **kwargs):
    # Validates cache directory exists; optionally copies to output_dir

def _split_generators(self, dl_manager):
    # Reads split info from cached dataset_info.json
    # Returns SplitGenerator for each split with Arrow shard file paths

def _generate_tables(self, files):
    # Yields (Key, pa.Table) for each record batch in cached Arrow files

Import

# Used internally by load_dataset when loading from cache
from datasets import load_dataset
ds = load_dataset("dataset_name")  # loads from cache if available

I/O Contract

Cache.__init__ Inputs

Name	Type	Required	Description
cache_dir	Optional[str]	No	Path to the cache directory. Defaults to HF_DATASETS_CACHE.
dataset_name	Optional[str]	Conditional	Name of the dataset. Required if repo_id is not provided.
config_name	Optional[str]	No	Configuration name to load from cache.
version	Optional[str]	No	Dataset version. Set to "auto" to resolve from cache. Defaults to "0.0.0".
hash	Optional[str]	No	Cache hash. Set to "auto" to resolve from cache.
base_path	Optional[str]	No	Base path for relative file resolution.
info	Optional[DatasetInfo]	No	Pre-existing dataset info object.
features	Optional[Features]	No	Custom features for cache lookup matching.
token	Optional[Union[bool, str]]	No	Authentication token.
repo_id	Optional[str]	Conditional	Repository ID. Required if dataset_name is not provided.
data_files	Optional[Union[str, list, dict, DataFilesDict]]	No	Data files specification (forwarded to config_kwargs).
data_dir	Optional[str]	No	Data directory (forwarded to config_kwargs).
storage_options	Optional[dict]	No	Storage backend options.
writer_batch_size	Optional[int]	No	Batch size for writing.
**config_kwargs		No	Additional configuration keyword arguments.

_find_hash_in_cache Inputs

Name	Type	Required	Description
dataset_name	str	Yes	Name of the dataset to locate in cache.
config_name	Optional[str]	No	Specific configuration name to search for.
cache_dir	Optional[str]	No	Cache directory root. Defaults to HF_DATASETS_CACHE.
config_kwargs	dict	Yes	Additional config keyword arguments used for config ID computation.
custom_features	Optional[Features]	No	Custom features used for config ID computation.

Outputs

Name	Type	Description
(from _find_hash_in_cache)	tuple[str, str, str]	A tuple of (config_name, version, hash) identifying the cache location.
(from _generate_tables)	tuple[Key, pa.Table]	Yields tuples of (Key(file_idx, batch_idx), pa_table) for each record batch in each cached Arrow shard.

Usage Examples

Loading from Cache

from datasets import load_dataset

# When a dataset has been previously downloaded and processed,
# load_dataset will use the Cache builder automatically
ds = load_dataset("rotten_tomatoes")

# Force loading from cache with auto-resolution
from datasets.packaged_modules.cache.cache import Cache
builder = Cache(
    dataset_name="rotten_tomatoes",
    version="auto",
    hash="auto",
)
builder.download_and_prepare()
ds = builder.as_dataset()

Finding Cache Location

from datasets.packaged_modules.cache.cache import _find_hash_in_cache

config_name, version, hash = _find_hash_in_cache(
    dataset_name="rotten_tomatoes",
    config_name=None,
    cache_dir=None,  # uses default HF_DATASETS_CACHE
    config_kwargs={},
    custom_features=None,
)
print(f"Config: {config_name}, Version: {version}, Hash: {hash}")

Related Pages

Implements Principle

• Principle:Huggingface_Datasets_Cache_Dataset_Building