Principle:Promptfoo Promptfoo CI Pipeline Configuration
| Knowledge Sources | |
|---|---|
| Domains | CI_CD, DevOps |
| Last Updated | 2026-02-14 08:00 GMT |
Overview
A CI/CD integration pattern for embedding LLM evaluations and security scans as automated quality gates in continuous integration pipelines.
Description
CI Pipeline Configuration is the practice of running Promptfoo evaluations and red team scans as part of CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins, Azure Pipelines). This enables automated regression testing of LLM applications on every code change.
The pattern leverages the `promptfoo eval` and `promptfoo redteam run` CLI commands, which produce exit codes that CI systems interpret as pass/fail gates. This ensures that LLM quality and security regressions are caught before deployment.
Usage
Use this principle when setting up automated LLM testing in CI/CD pipelines. It applies to any CI system that supports running Node.js commands and interpreting exit codes.
Theoretical Basis
Pseudo-code Logic:
1. Install promptfoo (npx or npm install)
2. Configure environment (API keys as CI secrets)
3. Run evaluation: npx promptfoo eval -c config.yaml --no-cache
4. CI interprets exit code: 0 = pass, 1 = failures detected
5. Optionally output artifacts: -o results.json
6. Optionally post results as PR comments (GitHub Action)