Workflow:Promptfoo Promptfoo Custom Provider Integration
| Knowledge Sources | |
|---|---|
| Domains | LLM_Ops, Integration, Extensibility |
| Last Updated | 2026-02-14 08:00 GMT |
Overview
End-to-end process for integrating a custom LLM endpoint, RAG pipeline, or agent system into Promptfoo's evaluation framework using built-in provider types or custom implementations.
Description
This workflow covers the process of connecting a non-standard LLM application to Promptfoo for evaluation and red teaming. Promptfoo supports multiple integration paths: HTTP providers for REST APIs, Python providers for Python-based applications, JavaScript/TypeScript providers for Node.js applications, and custom script providers for any executable. Each path implements the common ApiProvider interface, allowing the custom target to participate in all Promptfoo features including evaluations, red teaming, and CI/CD pipelines.
Usage
Execute this workflow when you need to:
- Evaluate a custom LLM application hosted behind a REST API
- Test a RAG pipeline that requires custom retrieval logic
- Red team an agent system with tool-calling capabilities
- Integrate a proprietary or self-hosted model not covered by built-in providers
- Connect a complex multi-step LLM pipeline as a single evaluation target
Input state: A running LLM application (API endpoint, Python script, or custom service) that accepts prompts and returns responses.
Output state: A Promptfoo provider configuration that can be referenced in any promptfooconfig.yaml for evaluation or red teaming.
Execution Steps
Step 1: Provider Type Selection
Choose the appropriate provider integration type based on your application's architecture. HTTP providers are best for applications exposed as REST APIs. Python providers are ideal for Python ML pipelines where you need direct access to the runtime. JavaScript/TypeScript providers offer native integration with Promptfoo's Node.js runtime. Custom script providers work with any executable that can read stdin and write to stdout.
Key considerations:
- HTTP providers require the least code; just configure URL, headers, and body template
- Python providers run in a subprocess and communicate via JSON serialization
- JavaScript providers run in-process and have access to the full Promptfoo context
- Consider using the HTTP provider with response transforms for complex API responses
Step 2: Interface Implementation
Implement the provider interface according to the chosen type. For HTTP providers, define the request URL, method, headers, and body template with the Template:Prompt placeholder. For Python providers, implement a call_api function that accepts a prompt string and returns an output string. For JavaScript providers, implement a module exporting an ApiProvider class with a callApi method.
Key considerations:
- The ApiProvider interface requires at minimum an id method and a callApi method
- callApi receives the rendered prompt and a CallApiContextParams object with variables and test context
- Return objects must include output (string or object) and optionally tokenUsage and cost
- Error handling should return an error field rather than throwing exceptions
Step 3: Request and Response Configuration
Configure how prompts are sent to the target and how responses are extracted. For HTTP providers, define request body templates, headers (including authentication), and response body transforms using JSONPath or JavaScript expressions. For code-based providers, handle the prompt-to-request and response-to-output mapping in the implementation code.
Key considerations:
- HTTP body templates support Nunjucks syntax for dynamic request construction
- Response transforms can extract specific fields from complex JSON responses
- Session management for stateful applications (e.g., multi-turn conversations) requires custom state tracking
- Authentication tokens can be injected via environment variables in headers
Step 4: Configuration Registration
Register the custom provider in the Promptfoo configuration file. Add the provider to the providers array using the appropriate prefix (file://, python://, or https://). Configure provider-level settings such as id (for display), label (for reports), env (for environment variables), and config (for provider-specific parameters).
Key considerations:
- File-based providers use file://path/to/provider.js or python://path/to/provider.py syntax
- HTTP providers can be defined inline or referenced from a YAML file
- Provider IDs should be unique and descriptive for clear reporting
- Provider-level transform functions can post-process responses before assertion evaluation
Step 5: Validation and Testing
Verify that the custom provider works correctly within the Promptfoo evaluation framework. Run a small evaluation with a few test cases to confirm that prompts are delivered correctly, responses are captured, and assertions can grade the output. Check for proper error handling, token usage reporting, and cleanup behavior.
Key considerations:
- Use --verbose flag to debug request/response details during initial testing
- Verify that the provider's output format is compatible with configured assertions
- Test edge cases: empty responses, timeouts, authentication failures, and rate limiting
- For red team targets, ensure the injectVar is correctly mapped to the adversarial input field