Principle:Promptfoo Promptfoo Configuration Loading
| Knowledge Sources | |
|---|---|
| Domains | Configuration, Evaluation |
| Last Updated | 2026-02-14 08:00 GMT |
Overview
A configuration resolution mechanism that loads, merges, dereferences, and validates YAML/JSON evaluation configurations into a fully resolved test suite.
Description
Configuration Loading is the process of transforming raw YAML or JSON configuration files into a complete, validated TestSuite object ready for evaluation execution. In Promptfoo, evaluation configurations define providers (LLMs to test), prompts (inputs to send), test cases (expected behaviors), and assertions (grading criteria).
The process solves the problem of managing complex, multi-file evaluation setups by supporting:
- JSON Schema $ref dereferencing: Inline external file references so a single config can compose from many sources.
- Multi-config merging: Combine multiple YAML/JSON configs into one unified evaluation specification.
- Default config discovery: Automatically find `promptfooconfig.yaml` in the working directory when no explicit path is given.
- Template rendering: Use Nunjucks templates in config values for dynamic configuration.
- CLI override merging: Apply command-line flags (filters, provider overrides) on top of file-based config.
Usage
Use this principle when initializing any evaluation run. Configuration Loading is the mandatory first step in both CLI-based (`promptfoo eval`) and programmatic (`evaluate()` API) evaluation flows. It converts user-authored config files into the internal TestSuite data structure that the evaluation engine requires.
Theoretical Basis
The configuration loading pipeline follows a layered resolution pattern:
Pseudo-code Logic:
1. Discover default config (promptfooconfig.yaml in cwd)
2. Load explicit config files from CLI --config flag
3. Parse YAML/JSON into raw UnifiedConfig objects
4. Dereference all JSON Schema $ref pointers
5. Merge multiple configs (providers, prompts, tests concatenated)
6. Apply CLI overrides (filters, provider restrictions)
7. Resolve relative file paths against config base directory
8. Validate and construct final TestSuite object
This layered approach ensures that file-based defaults can always be overridden by more specific CLI arguments, following the convention-over-configuration principle.