Principle:Openai Evals Model Based Classification
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, LLM_as_Judge |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
An evaluation pattern where an LLM acts as judge to classify model outputs using configurable prompts and scoring rubrics.
Description
Model Based Classification is the end-to-end evaluation pattern where a grading LLM evaluates the outputs of a subject model. For each sample, the subject model generates a completion, which is then formatted into an evaluation prompt and sent to the grading model. The grading model selects a classification choice (e.g. "Yes"/"No"), which is mapped to a numeric score. The eval aggregates scores across all samples and optionally supports meta-evaluation (comparing grading model judgments against ground-truth labels). This enables evaluation of open-ended tasks where exact matching is impossible.
Usage
Use model-based classification for evaluating tasks with open-ended outputs: factual accuracy, answer quality, helpfulness, and style. This is the primary mechanism for LLM-as-judge evaluations in the evals framework.
Theoretical Basis
The model-based classification pipeline:
- Generate — Subject model produces completion for each sample prompt
- Format — Completion and expected answer are inserted into grading prompt template
- Grade — Grading model reads formatted prompt and produces a verdict
- Parse — Verdict is parsed into a choice string using the configured match function
- Score — Choice string is mapped to a numeric score
- Aggregate — Scores are averaged across all samples
Meta-evaluation: When metaeval=True, the dataset includes ground-truth choice labels. The meta-score measures how often the grading model agrees with the ground truth, validating the grading model's reliability.