Implementation:Openai Evals MMLU Eval Config

Knowledge Sources	Openai_Evals
Domains	Evaluation, Configuration
Last Updated	2026-02-14 10:00 GMT

Overview

The MMLU (Massive Multitask Language Understanding) eval configuration file registers 57 subject-specific multiple-choice evaluations that measure a language model's knowledge across a broad range of academic and professional domains.

Description

mmlu.yaml is a declarative YAML configuration file located in the OpenAI Evals registry. It defines evaluation entries for the MMLU benchmark, originally introduced by Dan Hendrycks et al. The file contains 455 lines organized as 57 pairs of YAML mappings -- one alias entry and one versioned specification entry per subject.

Each subject follows a two-entry pattern:

Alias entry (e.g., mmlu-abstract-algebra) -- provides a human-friendly evaluation name, points to a specific versioned eval id, and declares [accuracy] as the metric.
Versioned entry (e.g., mmlu-abstract-algebra.val.ab-v1) -- specifies the eval class (evals.elsuite.multiple_choice:MultipleChoice) and loads the corresponding subject data from the HuggingFace hendrycks_test dataset using the hf:// URI scheme with the validation split.

The 57 subjects span the following broad categories:

STEM: abstract algebra, astronomy, college biology, college chemistry, college computer science, college mathematics, college physics, computer security, conceptual physics, electrical engineering, elementary mathematics, high school biology, high school chemistry, high school computer science, high school mathematics, high school physics, high school statistics, machine learning
Humanities: formal logic, high school european history, high school us history, high school world history, international law, jurisprudence, logical fallacies, moral disputes, moral scenarios, philosophy, prehistory, world religions
Social Sciences: econometrics, high school geography, high school government and politics, high school macroeconomics, high school microeconomics, high school psychology, public relations, security studies, sociology, us foreign policy
Professional: business ethics, clinical knowledge, college medicine, global facts, human aging, human sexuality, management, marketing, medical genetics, miscellaneous, nutrition, professional accounting, professional law, professional medicine, professional psychology, virology
Other: anatomy

All evaluations use the same eval class and differ only in the name query parameter of the HuggingFace dataset URL.

Usage

Use this configuration when you want to benchmark a model's factual knowledge and reasoning ability across 57 diverse academic subjects. This is the standard way to run MMLU evaluations within the OpenAI Evals framework. Each subject can be run independently by referencing its alias (e.g., mmlu-abstract-algebra), or the entire suite can be invoked via an eval set if one is defined.

Code Reference

Source Location

Repository: Openai_Evals
File: evals/registry/evals/mmlu.yaml
Lines: 1-455

Configuration Schema

The following shows the repeating pattern used for each of the 57 subjects:

# Alias entry -- human-friendly name
mmlu-abstract-algebra:
  id: mmlu-abstract-algebra.val.ab-v1
  metrics: [accuracy]

# Versioned entry -- eval class and dataset source
mmlu-abstract-algebra.val.ab-v1:
  class: evals.elsuite.multiple_choice:MultipleChoice
  args:
    dataset: hf://hendrycks_test?name=abstract_algebra&split=validation

A second representative example for a different subject category:

mmlu-high-school-european-history:
  id: mmlu-high-school-european-history.val.ab-v1
  metrics: [accuracy]

mmlu-high-school-european-history.val.ab-v1:
  class: evals.elsuite.multiple_choice:MultipleChoice
  args:
    dataset: hf://hendrycks_test?name=high_school_european_history&split=validation

I/O Contract

Inputs

Name	Type	Required	Description
id	string	Yes	Versioned eval identifier that the alias resolves to (e.g., mmlu-abstract-algebra.val.ab-v1)
metrics	list[string]	Yes	List of metric names to compute; always [accuracy] for MMLU
class	string	Yes	Fully-qualified Python class path (evals.elsuite.multiple_choice:MultipleChoice)
args.dataset	string (URI)	Yes	HuggingFace dataset URI in the form hf://hendrycks_test?name={subject}&split=validation

Outputs

Name	Type	Description
accuracy	float	Fraction of questions the model answered correctly (0.0 to 1.0) for the given subject

Usage Examples

Running a Single MMLU Subject

oaieval gpt-3.5-turbo mmlu-abstract-algebra

Running Another Subject

oaieval gpt-4 mmlu-professional-medicine

Running via the Versioned ID Directly

oaieval gpt-4 mmlu-college-physics.val.ab-v1

Related Pages

Openai_Evals_Eval_YAML_Registration -- describes the registry mechanism that loads this YAML file
Openai_Evals_Registry_Get_Eval -- the function that resolves eval aliases to versioned specs
Openai_Evals_Oaieval_Run -- the CLI entrypoint used to launch evaluations

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment