Implementation:CrewAIInc CrewAI Patronus Local Evaluator Tool
| Knowledge Sources | |
|---|---|
| Domains | Tools, Evaluation |
| Last Updated | 2026-02-11 00:00 GMT |
Overview
PatronusLocalEvaluatorTool evaluates model inputs and outputs using custom user-defined local evaluator functions registered with the Patronus client.
Description
PatronusLocalEvaluatorTool extends BaseTool and enables offline/custom evaluation logic by allowing developers to register their own Python evaluator functions with the Patronus client. During initialization, it checks for the patronus package availability (with an interactive auto-install fallback using uv add patronus), assigns the provided Patronus Client instance, and generates the tool description. The _run method extracts evaluation parameters from kwargs and calls self.client.evaluate() using the locally registered evaluator, returning a formatted string with the pass/fail result and explanation. A model_rebuild call at module level ensures the Pydantic model is properly initialized with the Client type.
Usage
Use this tool when you need domain-specific evaluation criteria that are not available as predefined Patronus evaluators. It is valuable for custom quality assurance pipelines where evaluation functions are implemented as local Python functions registered with the Patronus client.
Code Reference
Source Location
- Repository: CrewAI
- File: lib/crewai-tools/src/crewai_tools/tools/patronus_eval_tool/patronus_local_evaluator_tool.py
- Lines: 1-114
Signature
class FixedLocalEvaluatorToolSchema(BaseModel):
evaluated_model_input: str = Field(...)
evaluated_model_output: str = Field(...)
evaluated_model_retrieved_context: str = Field(...)
evaluated_model_gold_answer: str = Field(...)
evaluator: str = Field(...)
class PatronusLocalEvaluatorTool(BaseTool):
name: str = "Patronus Local Evaluator Tool"
args_schema: type[BaseModel] = FixedLocalEvaluatorToolSchema
client: Client = None
evaluator: str
evaluated_model_gold_answer: str
package_dependencies: list[str] # ["patronus"]
def __init__(self, patronus_client=None, evaluator="", evaluated_model_gold_answer="", **kwargs)
def _run(self, **kwargs) -> Any
Import
from crewai_tools import PatronusLocalEvaluatorTool
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| evaluated_model_input | str | Yes | The agent's task description in simple text |
| evaluated_model_output | str | Yes | The agent's output of the task |
| evaluated_model_retrieved_context | str | Yes | The agent's context |
| evaluated_model_gold_answer | str | Yes (at init) | The agent's gold answer, set at initialization |
| evaluator | str | Yes (at init) | The registered local evaluator name, set at initialization |
Outputs
| Name | Type | Description |
|---|---|---|
| _run() returns | str | Formatted string: "Evaluation result: {pass/fail}, Explanation: {explanation}" |
Usage Examples
Basic Usage
from crewai_tools import PatronusLocalEvaluatorTool
from patronus import Client
client = Client()
tool = PatronusLocalEvaluatorTool(
patronus_client=client,
evaluator="my_custom_evaluator",
evaluated_model_gold_answer="Expected answer"
)
result = tool._run(
evaluated_model_input="What is the capital of France?",
evaluated_model_output="Paris",
evaluated_model_retrieved_context="France is a country in Europe."
)