Implementation:Huggingface Datasets Extension To Module

Knowledge Sources	Huggingface Datasets HF Datasets Docs
Domains	Data_Engineering, NLP
Last Updated	2026-02-14 18:00 GMT

Overview

Concrete tool for mapping file extensions to dataset builder modules for automatic format detection, provided by the HuggingFace Datasets library.

Description

_EXTENSION_TO_MODULE is a module-level dictionary in datasets.packaged_modules that maps file extension strings to tuples of (module_name, default_kwargs). It is the primary registry used by the dataset loading pipeline to automatically determine which builder module should handle a given data file based on its extension. The dictionary is populated in two phases: first with core format mappings (CSV, JSON, Parquet, etc.), then dynamically extended with extensions from folder-based builders (ImageFolder, AudioFolder, VideoFolder, PdfFolder, NiftiFolder) in both lowercase and uppercase forms.

Usage

_EXTENSION_TO_MODULE is used internally by the dataset module factory during format inference. It is not typically called directly by end users, but understanding its structure is important for:

• Knowing which file formats are automatically supported.
• Debugging why a particular file extension is or is not recognized.
• Understanding how default builder kwargs (like {"sep": "\t"} for TSV files) are applied.

Code Reference

Source Location

• Repository: datasets
• File: src/datasets/packaged_modules/__init__.py
• Lines: 73-101

Signature

_EXTENSION_TO_MODULE: dict[str, tuple[str, dict]] = {
    ".csv": ("csv", {}),
    ".tsv": ("csv", {"sep": "\t"}),
    ".json": ("json", {}),
    ".jsonl": ("json", {}),
    ".ndjson": ("json", {}),
    ".parquet": ("parquet", {}),
    ".geoparquet": ("parquet", {}),
    ".gpq": ("parquet", {}),
    ".arrow": ("arrow", {}),
    ".txt": ("text", {}),
    ".tar": ("webdataset", {}),
    ".xml": ("xml", {}),
    ".hdf5": ("hdf5", {}),
    ".h5": ("hdf5", {}),
    ".eval": ("eval", {}),
    ".lance": ("lance", {}),
}
# Dynamically extended with:
_EXTENSION_TO_MODULE.update({ext: ("imagefolder", {}) for ext in imagefolder.ImageFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext.upper(): ("imagefolder", {}) for ext in imagefolder.ImageFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext: ("audiofolder", {}) for ext in audiofolder.AudioFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext.upper(): ("audiofolder", {}) for ext in audiofolder.AudioFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext: ("videofolder", {}) for ext in videofolder.VideoFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext.upper(): ("videofolder", {}) for ext in videofolder.VideoFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext: ("pdffolder", {}) for ext in pdffolder.PdfFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext.upper(): ("pdffolder", {}) for ext in pdffolder.PdfFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext: ("niftifolder", {}) for ext in niftifolder.NiftiFolder.EXTENSIONS})
_EXTENSION_TO_MODULE.update({ext.upper(): ("niftifolder", {}) for ext in niftifolder.NiftiFolder.EXTENSIONS})

Import

from datasets.packaged_modules import _EXTENSION_TO_MODULE

I/O Contract

Inputs

Name	Type	Required	Description
key (extension)	str	Yes	A file extension string including the leading dot (e.g. ".csv", ".jsonl", ".parquet").

Outputs

Name	Type	Description
value (module_name, default_kwargs)	tuple[str, dict]	A tuple where the first element is the packaged module name (e.g. "csv", "json", "parquet") and the second is a dictionary of default keyword arguments to pass to the builder (e.g. {"sep": "\t"} for TSV files).

Core Extension Mappings

Extension	Module	Default kwargs
.csv	csv	{}
.tsv	csv	{"sep": "\t"}
.json	json	{}
.jsonl	json	{}
.ndjson	json	{}
.parquet	parquet	{}
.geoparquet	parquet	{}
.gpq	parquet	{}
.arrow	arrow	{}
.txt	text	{}
.tar	webdataset	{}
.xml	xml	{}
.hdf5 / .h5	hdf5	{}
.eval	eval	{}
.lance	lance	{}

Additionally, image extensions (e.g. .jpg, .png), audio extensions (e.g. .wav, .mp3), video extensions, PDF extensions, and NIfTI extensions are dynamically registered from their respective folder builder classes.

Usage Examples

Basic Usage

from datasets.packaged_modules import _EXTENSION_TO_MODULE

# Look up the module for a CSV file
module_name, kwargs = _EXTENSION_TO_MODULE[".csv"]
print(module_name)  # "csv"
print(kwargs)       # {}

# Look up the module for a TSV file (note the default separator)
module_name, kwargs = _EXTENSION_TO_MODULE[".tsv"]
print(module_name)  # "csv"
print(kwargs)       # {"sep": "\t"}

# Check if an extension is supported
if ".parquet" in _EXTENSION_TO_MODULE:
    print("Parquet format is supported")

Listing All Supported Extensions

from datasets.packaged_modules import _EXTENSION_TO_MODULE

# Print all supported extensions grouped by module
from collections import defaultdict
module_to_exts = defaultdict(list)
for ext, (module, _) in _EXTENSION_TO_MODULE.items():
    module_to_exts[module].append(ext)

for module, exts in sorted(module_to_exts.items()):
    print(f"{module}: {sorted(exts)}")

Related Pages

Implements Principle

• Principle:Huggingface_Datasets_Extension_Module_Mapping