Principle:Openai Evals Model Graded Spec Design
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, LLM_as_Judge |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A specification pattern that defines how an LLM should evaluate model outputs by configuring evaluation prompts, valid choice strings, and scoring rubrics.
Description
Model Graded Spec Design is the process of creating a ModelGradedSpec that configures an LLM-as-judge evaluation. The spec defines: the evaluation prompt template (with placeholders for dynamic content), the valid choice strings the grading model must select from (e.g. "Yes"/"No" or "A"/"B"/"C"), a mapping of sample fields to template placeholders, and optional numeric scores per choice. Specs are stored as YAML files in evals/registry/modelgraded/ and referenced by ModelBasedClassify evaluations. Pre-built specs exist for factual accuracy (fact), closed-QA (closedqa), and head-to-head comparison (battle).
Usage
Design a model-graded spec when creating an evaluation where human-like judgment is needed to assess open-ended model outputs. This includes factual accuracy assessment, quality comparison, style evaluation, and other tasks where exact string matching is insufficient.
Theoretical Basis
A model-graded specification consists of:
- prompt — Template string or chat prompt with {placeholders} for dynamic content
- choice_strings — Valid answers the grading model can produce (e.g. ["Yes", "No", "Unsure"])
- input_outputs — Mapping of sample keys to template variables (e.g. {"input": "completion", "ideal": "expected"})
- eval_type — Classification strategy: "classify", "classify_cot", "cot_classify"
- choice_scores — Optional numeric scores per choice (e.g. {"Yes": 1.0, "No": 0.0})
Example fact-checking spec:
prompt: "You are comparing a submitted answer to an expert answer. Is the submission correct? {input} Expert: {ideal} Submitted: {completion}"
choice_strings: ["Yes", "No", "Unsure"]
input_outputs:
input: completion
ideal: expected
eval_type: cot_classify
choice_scores:
"Yes": 1.0
"No": 0.0
"Unsure": 0.5