Principle:Openai Evals Dataset Preparation
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Data_Engineering |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A data formatting and loading convention that standardizes evaluation datasets as JSONL files with required input-output field structures.
Description
Dataset Preparation defines the contract between evaluation data and the eval engine. Datasets are stored as JSONL (JSON Lines) files where each line is a self-contained JSON object representing one evaluation sample. The standard format requires an "input" field (the prompt, either a string or chat message list) and an "ideal" field (the expected answer, string or list of acceptable strings). The data loading system supports local files, remote URLs via blobfile, and compressed formats (gzip, lz4, zstandard). Datasets are placed in evals/registry/data/ by convention.
Usage
Prepare datasets before registering a new eval or when creating custom evaluation tasks. Any eval that extends the base Eval class and uses get_samples() expects data in this JSONL format.
Theoretical Basis
The JSONL format provides key advantages for evaluation:
- Line independence — Each sample is self-contained, enabling streaming and parallel processing
- Flexible schema — Different eval types can require different fields
- Compression support — Large datasets can use .gz, .lz4, or .zst extensions
- Directory recursion — A path pointing to a directory loads all .jsonl files within it
Standard sample format:
{"input": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is 2+2?"}], "ideal": "4"}
Model-graded sample format:
{"input": "What is the capital of France?", "ideal": "Paris", "completion": "The capital is Paris."}