Implementation:Openai Evals Progress Class

Knowledge Sources	OpenAI Evals
Domains	Evaluation, Reliability
Last Updated	2026-02-14 10:00 GMT

Overview

Concrete tool for tracking batch evaluation progress with file-based persistence provided by the oaievalset module.

Description

The Progress class manages a JSONL progress file that records completed eval commands. It provides load() to read existing progress, add() to append a completed command (which auto-saves), and save() to persist the current state. The file is stored at /tmp/oaievalset/{model}.{eval_set}.progress.txt by default.

Usage

Used internally by oaievalset.run() to track which evals in a set have been completed. Enables resume functionality for interrupted batch runs.

Code Reference

Source Location

• Repository: openai/evals
• File: evals/cli/oaievalset.py (lines 17-40)

Signature

class Progress:
    def __init__(self, file: str) -> None:
        """
        Args:
            file: Path to progress file (JSONL format).
        """

    def load(self) -> bool:
        """Load progress from file. Returns True if any progress was loaded."""

    def add(self, item: Task) -> None:
        """Add a completed task and save immediately."""

    def save(self) -> None:
        """Write all completed tasks to the progress file."""

Import

from evals.cli.oaievalset import Progress

I/O Contract

Inputs

Name	Type	Required	Description
file	str	Yes	Path to progress file
item	list[str]	Yes (for add)	Completed command as list of argument strings

Outputs

Name	Type	Description
completed	list[list[str]]	List of completed command argument lists
Progress file	JSONL	One JSON array per line, each representing a completed command

Usage Examples

Tracking Progress

from evals.cli.oaievalset import Progress

progress = Progress("/tmp/oaievalset/gpt-4.test-basic.progress.txt")

# Load existing progress
if progress.load():
    print(f"Resuming: {len(progress.completed)} evals already done")

# Mark a command as completed
progress.add(["oaieval", "gpt-4", "test-match"])

# Check completed list
print(progress.completed)

Related Pages

Implements Principle

• Principle:Openai_Evals_Eval_Progress_Tracking

Uses Heuristic

• Heuristic:Openai_Evals_Eval_Resumption_Strategy