Principle:Promptfoo Promptfoo Test Suite Construction
| Knowledge Sources | |
|---|---|
| Domains | Testing, Evaluation |
| Last Updated | 2026-02-14 08:00 GMT |
Overview
A test case assembly mechanism that loads prompts, test cases, and filters from multiple source formats into a structured evaluation suite.
Description
Test Suite Construction is the process of assembling the complete set of evaluation inputs from heterogeneous sources. In LLM evaluation, test cases define the specific input-output scenarios to validate, while prompts define the templates and filters customize output processing.
The challenge this solves is that test data comes in many formats: inline YAML objects, CSV files, JSON arrays, glob patterns, and even programmatic functions. Test Suite Construction normalizes all these into a uniform TestCase[] array with consistent vars, assert, and options fields.
Usage
Use this principle when preparing evaluation data from diverse sources. This is the third step in the evaluation pipeline, after configuration loading and provider resolution. It is necessary whenever test cases come from external files or need programmatic generation.
Theoretical Basis
Pseudo-code Logic:
1. Process prompts: convert strings, file paths, functions → Prompt[]
2. Read tests: for each test source:
a. If string path → load from CSV/YAML/JSON/glob
b. If object with path → load file at path
c. If inline object → use directly
d. Flatten nested arrays into flat TestCase[]
3. Read filters: load Nunjucks filter functions from file paths
4. Combine into TestSuite structure