Implementation:Promptfoo Promptfoo resolveConfigs
| Knowledge Sources | |
|---|---|
| Domains | Configuration, Evaluation |
| Last Updated | 2026-02-14 08:00 GMT |
Overview
Concrete tool for resolving evaluation configurations from CLI options and file-based configs into a complete TestSuite, provided by the Promptfoo framework.
Description
The resolveConfigs function is the primary entry point for configuration loading in Promptfoo. It takes CLI options and a default config, then orchestrates the full resolution pipeline: loading config files, dereferencing `$ref` pointers, merging multiple configs, applying CLI overrides, and constructing the final TestSuite object. It also delegates to dereferenceConfig for JSON Schema ref resolution and loadDefaultConfig for auto-discovery of config files.
Usage
Import this function when building evaluation pipelines that need to resolve user-provided configuration into executable test suites. This is called at the start of every `promptfoo eval` command and by any programmatic evaluation flow.
Code Reference
Source Location
- Repository: promptfoo
- File: src/util/config/load.ts
- Lines: L559-871
Signature
export async function resolveConfigs(
cmdObj: Partial<CommandLineOptions>,
_defaultConfig: Partial<UnifiedConfig>,
type?: 'DatasetGeneration' | 'AssertionGeneration',
): Promise<{
testSuite: TestSuite;
config: Partial<UnifiedConfig>;
basePath: string;
commandLineOptions?: Partial<CommandLineOptions>;
}>
Import
import { resolveConfigs } from './util/config/load';
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| cmdObj | Partial<CommandLineOptions> | Yes | CLI options including config file paths, filters, provider overrides |
| _defaultConfig | Partial<UnifiedConfig> | Yes | Base config to merge with (from loadDefaultConfig or empty) |
| type | 'DatasetGeneration' or 'AssertionGeneration' | No | Optional generation type for specialized validation |
Outputs
| Name | Type | Description |
|---|---|---|
| testSuite | TestSuite | Fully resolved test suite with providers, prompts, tests, assertions |
| config | Partial<UnifiedConfig> | The merged raw config (for reference) |
| basePath | string | Base directory for resolving relative file paths |
| commandLineOptions | Partial<CommandLineOptions> | Processed CLI options (optional) |
Usage Examples
CLI Eval Flow
import { resolveConfigs } from './util/config/load';
import { loadDefaultConfig } from './util/config/default';
// 1. Load default config from working directory
const { defaultConfig, defaultConfigPath } = await loadDefaultConfig();
// 2. Resolve with CLI options
const { testSuite, config, basePath } = await resolveConfigs(
{
config: ['./promptfooconfig.yaml'],
filterProviders: 'openai:*',
},
defaultConfig,
);
// 3. testSuite is ready for evaluation
console.log(`Resolved ${testSuite.providers.length} providers`);
console.log(`Resolved ${testSuite.tests?.length ?? 0} test cases`);