Implementation:CrewAIInc CrewAI Patronus Predefined Eval Tool
| Knowledge Sources | |
|---|---|
| Domains | Tools, Evaluation |
| Last Updated | 2026-02-11 00:00 GMT |
Overview
PatronusPredefinedCriteriaEvalTool evaluates model inputs and outputs using developer-specified, fixed evaluator/criteria pairs that the agent cannot change.
Description
PatronusPredefinedCriteriaEvalTool extends BaseTool and is the most controlled variant of the Patronus evaluation tools. It accepts a list of evaluator/criteria dictionaries during initialization (e.g., [{"evaluator": "judge", "criteria": "contains-code"}]), which are stored and embedded into the tool description. The _run method extracts evaluation parameters from kwargs, handles both string and dict inputs for each field (extracting the "description" key from dicts), uses the fixed self.evaluators list (ignoring any agent-provided evaluators), and POSTs to the Patronus evaluate endpoint with the PATRONUS_API_KEY header. It raises an exception on non-200 responses.
Usage
Use this tool for production workflows where evaluation consistency and predictability are important. The developer specifies exactly which evaluators and criteria should be used, ensuring the agent always uses the intended evaluation configuration.
Code Reference
Source Location
- Repository: CrewAI
- File: lib/crewai-tools/src/crewai_tools/tools/patronus_eval_tool/patronus_predefined_criteria_eval_tool.py
- Lines: 1-105
Signature
class FixedBaseToolSchema(BaseModel):
evaluated_model_input: dict = Field(...)
evaluated_model_output: dict = Field(...)
evaluated_model_retrieved_context: dict = Field(...)
evaluated_model_gold_answer: dict = Field(...)
evaluators: list[dict[str, str]] = Field(...)
class PatronusPredefinedCriteriaEvalTool(BaseTool):
name: str = "Call Patronus API tool for evaluation of model inputs and outputs"
evaluate_url: str = "https://api.patronus.ai/v1/evaluate"
args_schema: type[BaseModel] = FixedBaseToolSchema
evaluators: list[dict[str, str]] = Field(default_factory=list)
def __init__(self, evaluators: list[dict[str, str]], **kwargs)
def _run(self, **kwargs) -> Any
Import
from crewai_tools import PatronusPredefinedCriteriaEvalTool
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| evaluated_model_input | dict or str | Yes | The agent's task description |
| evaluated_model_output | dict or str | Yes | The agent's output of the task |
| evaluated_model_retrieved_context | dict or str | Yes | The agent's context |
| evaluated_model_gold_answer | dict or str | Yes | The agent's gold answer if available |
| evaluators | list[dict[str, str]] | Yes (at init) | Predefined evaluator/criteria pairs set at initialization |
Outputs
| Name | Type | Description |
|---|---|---|
| _run() returns | dict | JSON response from the Patronus Evaluation API containing evaluation results |
Usage Examples
Basic Usage
from crewai_tools import PatronusPredefinedCriteriaEvalTool
tool = PatronusPredefinedCriteriaEvalTool(
evaluators=[
{"evaluator": "judge", "criteria": "patronus:is-concise"},
{"evaluator": "judge", "criteria": "patronus:is-helpful"}
]
)
result = tool._run(
evaluated_model_input="Summarize the document",
evaluated_model_output="The document discusses...",
evaluated_model_retrieved_context="Full document text here",
evaluated_model_gold_answer="Expected summary"
)