Principle:Huggingface Datasets Hub Metadata Configs
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, NLP |
| Last Updated | 2026-02-14 18:00 GMT |
Overview
Managing YAML front matter metadata for dataset cards enables the Hub to discover dataset configurations, data files, and split information without downloading the data.
Description
Hub metadata configs manage the YAML front matter section of a dataset card (README.md), which tells the Hub how to locate and organize data files for each configuration (subset) of a dataset. Each configuration entry specifies a config_name, optional data_files patterns (with split-to-path mappings), and other configuration parameters. This metadata is critical for the Hub's auto-loading mechanism: when a user calls load_dataset, the Hub reads this YAML to determine which files belong to which configuration and split without scanning the entire repository. The metadata also controls which configuration is treated as the default.
Usage
Use Hub metadata configs when publishing datasets with multiple configurations (e.g., different languages, tasks, or versions) or when you need fine-grained control over how data files are mapped to splits. The metadata is automatically managed by push_to_hub but can also be manually edited in the dataset card.
Theoretical Basis
The metadata config system implements a declarative data-file-to-split mapping. Each config entry can specify data_files as a string (single file), a list of strings (multiple files for one split), or a list of split-path dictionaries (explicit split assignments). This flexibility accommodates datasets ranging from a single CSV file to complex multi-split, multi-configuration repositories. The system supports glob patterns for file matching, enabling dynamic file discovery. Validation ensures that split names conform to the allowed pattern and that data_files entries are well-formed. The default config is determined by explicit default: true flags, the name "default", or being the only configuration.