Principle:Huggingface Datasets Dataset Integrity Verification
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, NLP |
| Last Updated | 2026-02-14 18:00 GMT |
Overview
Dataset Integrity Verification provides the low-level utility functions that perform actual checksum comparisons, split size validation, and file size computation to ensure that downloaded and generated datasets match their expected specifications.
Description
While the higher-level Principle:Data_Verification principle covers the VerificationMode enum that controls when verification occurs, this principle addresses the underlying utility functions that perform the actual verification work. These functions live in the info utilities module and provide the concrete mechanisms for detecting data corruption, incomplete downloads, and generation errors.
The verify_checksums function compares a dictionary of expected file checksums against a dictionary of recorded (actual) checksums. When mismatches are detected, it raises a NonMatchingChecksumError with details about which files failed verification. Similarly, verify_splits compares expected split names and example counts against the recorded values, raising NonMatchingSplitsSizesError when discrepancies are found. Both functions support an optional verification mode parameter that allows callers to skip checks when appropriate.
Supporting these verification functions, get_size_checksum_dict computes SHA-256 checksums and file sizes for a collection of local files, producing the recorded-checksum dictionaries that the verification functions consume. The is_small_dataset utility determines whether a dataset falls below a configurable size threshold, which can influence caching and processing decisions. Together, these utilities form the computational backbone of the library's data integrity system.
Usage
Use Dataset Integrity Verification when:
- You need to verify that downloaded files match their expected checksums after a download operation completes.
- You are validating that generated dataset splits contain the expected number of examples.
- You are computing checksums for a set of files to store as metadata for future verification.
- You need to determine whether a dataset qualifies as "small" for the purpose of applying different processing strategies.
- You are implementing a custom dataset builder and want to integrate with the library's standard verification pipeline.
Theoretical Basis
Integrity verification relies on cryptographic hash functions (specifically SHA-256) to detect any modification or corruption of data files. A cryptographic hash produces a fixed-size digest that is computationally infeasible to forge: even a single-bit change in the input produces a completely different hash value. By comparing the hash of a downloaded file against a pre-recorded expected hash, the system can detect corruption with extremely high confidence.
Split size verification operates on a simpler statistical basis: the number of examples in each split is a deterministic property of the generation process. Any deviation from the expected count indicates either a bug in the generation code, a change in the source data, or an incomplete generation run. Combining both checksum and split verification provides defense in depth, catching errors at both the raw-file level and the semantic-output level.