Workflow:Promptfoo Promptfoo Project Initialization
| Knowledge Sources | |
|---|---|
| Domains | LLM_Ops, Developer_Experience, Onboarding |
| Last Updated | 2026-02-14 08:00 GMT |
Overview
Interactive scaffolding process that generates a complete Promptfoo evaluation project with configuration files, provider scripts, and example test cases tailored to the user's chosen use case.
Description
This workflow implements the promptfoo init command, which guides users through an interactive setup wizard. The wizard collects the user's intent (compare models, test RAG pipelines, or evaluate agents), preferred programming language, and target LLM providers. Based on these selections, it generates a promptfooconfig.yaml file, provider implementation scripts, and supporting files. The generated project is immediately runnable with promptfoo eval, providing a zero-to-working starting point.
Usage
Execute this workflow when you are:
- Starting a new LLM evaluation project from scratch
- Setting up Promptfoo for the first time in a repository
- Creating a standardized evaluation template for a team
- Exploring Promptfoo's capabilities with a guided example
Input state: An empty or existing directory where you want to initialize a Promptfoo project.
Output state: A directory containing promptfooconfig.yaml, provider scripts, and optionally a README, ready to run with promptfoo eval.
Execution Steps
Step 1: Use Case Selection
Present the user with interactive prompts to choose their evaluation scenario. The wizard offers three primary paths: model comparison (compare outputs across multiple LLMs), RAG evaluation (test retrieval-augmented generation pipelines), and agent evaluation (test autonomous agent behavior). Each path generates different scaffolding tailored to the use case.
Key considerations:
- The --example flag can skip interactive prompts and use a predefined template
- Use case selection determines the prompt template, assertion types, and provider scripts generated
- RAG and agent paths generate additional context loader scripts
Step 2: Language Selection
Choose the programming language for custom provider and assertion scripts. Options include JavaScript/TypeScript, Python, and Bash. The selection determines which file types are generated for custom providers, transforms, and assertion functions.
Key considerations:
- Python is the most common choice for ML/AI teams
- JavaScript/TypeScript integrates natively with the Promptfoo runtime
- Bash scripts are supported for simple command-line integrations
Step 3: Provider Selection
Select the LLM providers to include in the generated configuration. The wizard presents a curated list of popular providers (OpenAI, Anthropic, Google, AWS Bedrock, Azure, Ollama, etc.) and records selections. Missing API key warnings are displayed when relevant environment variables are not set.
Key considerations:
- Multiple providers can be selected for side-by-side comparison
- Provider selections are recorded for telemetry (anonymous usage tracking)
- API key availability is checked and warnings issued for missing credentials
- Custom provider options generate skeleton implementation files
Step 4: File Generation
Generate all project files based on the collected selections. The generator creates provider implementation scripts (in the selected language), context helper files for RAG/agent use cases, and any supporting data files. File templates use Nunjucks rendering with the collected configuration values.
Key considerations:
- Generated files include inline comments explaining customization points
- RAG templates include example context retrieval functions
- Agent templates include tool-calling scaffolding
- Existing files are not overwritten without confirmation
Step 5: Configuration Rendering
Render the final promptfooconfig.yaml from a Nunjucks template, incorporating all selections: prompts, providers, test cases, and assertions. The generated configuration includes example test cases with realistic variables and basic assertions (LLM-rubric, contains, similarity checks).
Key considerations:
- The generated config is immediately runnable with promptfoo eval
- Example assertions demonstrate both deterministic and model-graded evaluation
- Default test cases provide a starting point for customization
- An optional README file is generated with quick-start instructions