Principle:Huggingface Datasets Split Definition
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, NLP |
| Last Updated | 2026-02-14 18:00 GMT |
Overview
Defining standard dataset split names (TRAIN, TEST, VALIDATION) establishes a shared vocabulary for dataset partitioning across ML workflows.
Description
Split definition provides a standardized enumeration of dataset partition names. The three primary splits are TRAIN (data used for model training), VALIDATION (data used for hyperparameter tuning and model selection during development), and TEST (data reserved for final evaluation and metric reporting). An additional ALL split represents the union of all defined splits. Custom split names can also be created for domain-specific needs. Using standard split names ensures interoperability between datasets, training frameworks, and evaluation tools, and enables automatic split detection when loading datasets from the Hub.
Usage
Use standard split definitions when creating, loading, or publishing datasets. Reference the constants (Split.TRAIN, Split.TEST, Split.VALIDATION) instead of raw strings to benefit from IDE autocompletion and type checking. Custom splits can be created with Split("custom_name").
Theoretical Basis
The split system is built on the NamedSplit class, which is a string subclass that supports algebraic composition (e.g., Split.TRAIN + Split.TEST). This composition capability allows requesting multiple splits as a single dataset. The standard three-split convention (train/validation/test) reflects established ML methodology: training data for parameter learning, validation data for model selection, and test data for unbiased evaluation. The ALL sentinel represents the complete dataset, useful for data exploration or when splits are not meaningful.