Implementation:CrewAIInc CrewAI Patronus Eval Tool

Knowledge Sources	CrewAI
Domains	Tools, Evaluation
Last Updated	2026-02-11 00:00 GMT

Overview

PatronusEvalTool enables CrewAI agents to dynamically select the best evaluator and criteria from all available options on the Patronus platform for evaluating model inputs and outputs.

Description

PatronusEvalTool extends BaseTool and implements the most flexible of the three Patronus evaluation tools. During initialization, it fetches all available evaluators from the Patronus API (filtering out deprecated ones and deduplicating by ID) and all evaluator criteria (extracting pass_criteria, rubrics, and descriptions). It issues a warning that the agent will autonomously select evaluators, recommending PatronusPredefinedCriteriaEvalTool for more controlled scenarios. The _generate_description method builds a dynamic tool description embedding all available criteria, instructing the agent to choose the most appropriate evaluator/criteria pair. The _run method normalizes the selected evaluators and POSTs to the Patronus evaluate endpoint.

Usage

Use this tool when evaluation needs are not known in advance and the agent should intelligently select evaluation criteria at runtime. For controlled production scenarios where evaluation consistency is important, use PatronusPredefinedCriteriaEvalTool instead.

Code Reference

Source Location

• Repository: CrewAI
• File: lib/crewai-tools/src/crewai_tools/tools/patronus_eval_tool/patronus_eval_tool.py
• Lines: 1-156

Signature

class PatronusEvalTool(BaseTool):
    name: str = "Patronus Evaluation Tool"
    evaluate_url: str = "https://api.patronus.ai/v1/evaluate"
    evaluators: list[dict[str, str]] = Field(default_factory=list)
    criteria: list[dict[str, str]] = Field(default_factory=list)
    description: str = ""
    env_vars: list[EnvVar]  # PATRONUS_API_KEY required

    def _run(
        self,
        evaluated_model_input: str | None,
        evaluated_model_output: str | None,
        evaluated_model_retrieved_context: str | None,
        evaluators: list[dict[str, str]],
    ) -> Any

Import

from crewai_tools import PatronusEvalTool

I/O Contract

Inputs

Name	Type	Required	Description
evaluated_model_input	str or None	Yes	The agent's task description in simple text
evaluated_model_output	str or None	Yes	The agent's output of the task
evaluated_model_retrieved_context	str or None	Yes	The agent's context
evaluators	list[dict[str, str]]	Yes	List of evaluator/criteria pairs, e.g. [{"evaluator": "Judge", "criteria": "patronus:is-code"}]

Outputs

Name	Type	Description
_run() returns	dict	JSON response from the Patronus Evaluation API containing evaluation results

Usage Examples

Basic Usage

from crewai_tools import PatronusEvalTool

# Agent dynamically selects evaluators from available options
tool = PatronusEvalTool()
result = tool._run(
    evaluated_model_input="Summarize the document",
    evaluated_model_output="The document discusses...",
    evaluated_model_retrieved_context="Full document text here",
    evaluators=[{"evaluator": "judge", "criteria": "patronus:is-concise"}]
)

Related Pages

• Principle:CrewAIInc_CrewAI_Built_In_Tool_Selection