Workflow:FlowiseAI Flowise Evaluation Pipeline
| Knowledge Sources | |
|---|---|
| Domains | LLM_Evaluation, Quality_Assurance, LLM_Ops |
| Last Updated | 2026-02-12 07:30 GMT |
Overview
End-to-end process for systematically evaluating chatflow and agent flow quality using datasets, automated evaluators, and LLM-based grading in Flowise.
Description
This workflow covers the complete evaluation pipeline in Flowise, which enables users to benchmark their chatflows and agent flows against test datasets. The pipeline involves creating structured datasets with input-output pairs, defining evaluators (text-matching, numeric thresholds, or LLM-based grading), running evaluation batches across one or more flows, and analyzing results through charts and detailed metrics. Evaluations support versioning to track quality over time. Three evaluator categories are available: text-based (substring matching, prefix checking), numeric (token counts, latency, response length), and LLM-based (custom prompt-driven scoring with structured output schemas).
Usage
Execute this workflow when you need to systematically assess the quality of a chatflow or agent flow against a known set of test cases. This applies when you want to measure response accuracy, validate that outputs contain expected content, enforce latency or token budget constraints, or use an LLM judge to score response quality. The output is a comprehensive evaluation report with pass rates, latency charts, token usage breakdowns, and per-row results.
Execution Steps
Step 1: Create Evaluation Dataset
Navigate to the Datasets page and click "Add New". Provide a dataset name and description. Then add rows to the dataset, where each row contains input variables (key-value pairs representing the conversation input) and an expected output (the gold-standard response). Rows can be added individually or bulk-imported from a CSV file.
Key considerations:
- Each dataset row has input variables and an expected output
- CSV import maps columns to variables automatically
- Rows can be reordered, edited, or deleted after creation
- Datasets are reusable across multiple evaluation runs
Step 2: Define Evaluators
Navigate to the Evaluators page and create evaluators that define pass/fail criteria. Three evaluator types are available:
Text-based evaluators check string patterns in responses:
- ContainsAny, ContainsAll, DoesNotContainAny, DoesNotContainAll
- StartsWith, NotStartsWith
Numeric evaluators check metric thresholds:
- totalTokens, promptTokens, completionTokens (with operators: equals, greaterThan, lessThan, etc.)
- apiLatency, llm latency, chain latency
- responseLength
LLM-based evaluators use a language model to grade responses:
- Custom evaluation prompt with structured JSON output schema
- Properties with types, descriptions, and required flags
- Predefined sample prompts for common evaluation patterns
Key considerations:
- Multiple evaluators can be applied to a single evaluation run
- Text evaluators compare against comma-separated expected values
- Numeric evaluators use comparison operators against thresholds
- LLM evaluators require a chat model credential for the grading LLM
Step 3: Create Evaluation Run
Click "Create Evaluation" to open the multi-step wizard. In step one, select one or more chatflows or agent flows to evaluate and choose the dataset. In step two, select which simple evaluators (text and numeric) to apply. In step three, optionally enable LLM grading by selecting a chat model, providing credentials, and choosing LLM-based evaluators.
Key considerations:
- Multiple flows can be evaluated against the same dataset in one run
- The "dataset as one conversation" option treats all rows as a continuous dialog
- Evaluation name helps identify runs in the listing
- LLM grading requires a separate credential for the grading model (distinct from the chatflow's model)
Step 4: Run Evaluation
Submit the evaluation to begin execution. The system iterates through each dataset row, sends the input to the selected chatflow(s), captures the response along with metrics (tokens, latency), and applies all configured evaluators to determine pass/fail status. Results are computed asynchronously and stored with full audit trail.
Key considerations:
- Each dataset row is sent as a prediction request to the chatflow
- Metrics are captured for every request: tokens, latency, response length
- Text and numeric evaluators run locally; LLM evaluators invoke the grading model
- Failed rows capture error messages for debugging
Step 5: Analyze Results
Navigate to the evaluation result page to view comprehensive metrics. The dashboard displays a pass rate pie chart, latency distribution line chart, token usage stacked bar chart, and per-evaluator statistics. A detailed results table shows each dataset row with its actual output, expected output, pass/fail status per evaluator, and captured metrics.
Key considerations:
- Charts provide visual overview of quality, performance, and cost
- Per-row drill-down shows full input, output, and evaluator verdicts
- Side drawer provides expanded detail view for individual results
- Results can be filtered and sorted by evaluator or metric
Step 6: Iterate and Compare Versions
After making improvements to the chatflow, re-run the evaluation to generate a new version. The evaluation versions drawer shows all historical runs, enabling comparison of pass rates, latency, and token usage across versions. The system also detects when a chatflow has changed since the last evaluation.
Key considerations:
- Each re-run creates a new version linked to the same evaluation
- Version history enables tracking quality improvements over time
- Outdated evaluation detection alerts when the underlying flow has changed
- Evaluations can be deleted individually or in bulk