Implementation:Promptfoo Promptfoo Config Schema

Knowledge Sources	Promptfoo_Promptfoo
Domains	Configuration, Validation
Last Updated	2026-02-14 07:45 GMT

Overview

Concrete JSON Schema (Draft-07) definition that validates promptfoo configuration files (promptfooconfig.yaml), defining the complete structure of providers, test cases, assertions, and evaluation settings.

Description

The Config_Schema (config-schema.json) is a comprehensive JSON Schema that serves as the machine-readable specification for all valid promptfoo configuration files. It defines the UnifiedConfig structure including providers (70+ LLM backends), test cases, assertions (40+ types), prompts, output formats, red team configuration, and evaluation parameters. The schema is auto-generated from the Zod-based UnifiedConfigSchema in the source code using scripts/generateJsonSchema.ts.

Usage

Reference this schema in IDE settings (e.g., VS Code YAML extension) for autocomplete and validation when editing promptfooconfig.yaml files. It is also hosted on the documentation site for public consumption.

Code Reference

Source Location

• Repository: Promptfoo_Promptfoo
• File: site/static/config-schema.json
• Lines: 1-2991

Signature

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$ref": "#/definitions/UnifiedConfig",
  "definitions": {
    "UnifiedConfig": { /* ... */ },
    "TestCase": { /* ... */ },
    "Assertion": { /* ... */ }
  }
}

Import

# In promptfooconfig.yaml, reference the schema:
# yaml-language-server: $schema=https://www.promptfoo.dev/config-schema.json

I/O Contract

Inputs

Name	Type	Required	Description
(none)	—	—	Static schema consumed by validators

Outputs

Name	Type	Description
schema	JSON Schema	Validation rules for promptfoo configuration

Usage Examples

# promptfooconfig.yaml with schema reference
# yaml-language-server: $schema=https://www.promptfoo.dev/config-schema.json

description: My evaluation
providers:
  - openai:gpt-4o
prompts:
  - "Answer: {{question}}"
tests:
  - vars:
      question: "What is 2+2?"
    assert:
      - type: contains
        value: "4"

Related Pages

• Environment:Promptfoo_Promptfoo_Node_Runtime