Implementation:Huggingface Datasets Eval Builder

Knowledge Sources	Huggingface Datasets HF Datasets Docs
Domains	Data_Loading, Evaluation
Last Updated	2026-02-14 18:00 GMT

Overview

Packaged dataset builder for loading evaluation log data, provided by the HuggingFace Datasets library.

Description

Eval is a packaged dataset builder extending datasets.GeneratorBasedBuilder (rather than ArrowBasedBuilder) that loads evaluation log files containing sample-level results. Unlike other packaged builders, it processes structured log directories where each sample is stored as a separate JSON file within a samples/ subdirectory.

The builder infers features automatically by reading the first 5 examples (NUM_EXAMPLES_FOR_FEATURES_INFERENCE = 5), converting them to PyArrow tables, concatenating with schema promotion, and extracting the unified schema. During feature inference, dictionary values are serialized to JSON strings, and list values have each element serialized to JSON strings.

Sample files follow the naming convention {sample_idx}_epoch_{epoch_idx}.ext, and the builder sorts them by (epoch_idx, sample_idx) to ensure deterministic ordering. The _generate_examples method yields each sample as a dictionary, with dict-valued and list-valued fields serialized to JSON strings for Arrow compatibility.

The builder enables on-the-fly extraction for compressed log archives via dl_manager.download_config.extract_on_the_fly = True.

Usage

Use this builder via load_dataset("eval", data_files=...) to load evaluation log directories. It is also triggered automatically when files with the .eval extension are detected by the dataset loading pipeline.

Code Reference

Source Location

• Repository: datasets
• File: src/datasets/packaged_modules/eval/eval.py
• Lines: 1-77

Signature

class Eval(datasets.GeneratorBasedBuilder):
    NUM_EXAMPLES_FOR_FEATURES_INFERENCE = 5

Key methods:

def _info(self):
    return datasets.DatasetInfo()

def _split_generators(self, dl_manager):
    # Downloads and extracts log files
    # Infers features from first 5 examples if not already set
    # Returns SplitGenerator for each split

def _sort_samples_key(self, sample_path: str):
    # Parses "{sample_idx}_epoch_{epoch_idx}" from filename
    # Returns (epoch_idx, sample_idx) tuple for sorting

def _iter_samples_from_log_files(self, log_files: Iterable[str]):
    # Filters for files in "samples/" subdirectories
    # Sorts by epoch and sample index
    # Loads each JSON file and serializes nested dicts/lists to strings

def _generate_examples(self, base_files, logs_files_iterables):
    # Yields (Key(file_idx, sample_idx), sample_dict) for each sample

Import

# Used via load_dataset
from datasets import load_dataset
ds = load_dataset("eval", data_files="path/to/eval_logs")

I/O Contract

Inputs

Name	Type	Required	Description
data_files	str, list, or dict	Yes	Path(s) to evaluation log files or archives. Can be a single path, a list, or a dict mapping split names to file paths.

Expected Log Directory Structure

The builder expects log directories containing a samples/ subdirectory with individual JSON files:

eval_log/
  samples/
    0_epoch_0.json
    1_epoch_0.json
    0_epoch_1.json
    1_epoch_1.json

Each JSON file contains a dictionary representing one evaluation sample.

Outputs

Name	Type	Description
(from _generate_examples)	tuple[Key, dict]	Yields tuples of (Key(file_idx, sample_idx), sample_dict) for each sample in the log files.
(from load_dataset)	Dataset or DatasetDict	The loaded dataset with features inferred from the evaluation samples. Dict and list fields are serialized as JSON strings.

Usage Examples

Basic Usage

from datasets import load_dataset

# Load evaluation logs
ds = load_dataset("eval", data_files="eval_results/", split="train")
print(ds[0])
print(ds.features)

Loading from Archive

from datasets import load_dataset

# Load from a compressed eval log archive
ds = load_dataset("eval", data_files="eval_results.tar.gz", split="train")
for example in ds:
    print(example)

Related Pages

Implements Principle

• Principle:Huggingface_Datasets_Eval_Dataset_Building