Implementation:Run llama Llama index EvalQueryEngineTool

Overview

The EvalQueryEngineTool class extends QueryEngineTool by adding an evaluation step to every query engine response. After the underlying query engine produces a response, an evaluator (defaulting to AnswerRelevancyEvaluator) assesses the quality of the response. If the evaluation fails, the tool output is replaced with a failure message. This enables agents to automatically detect and handle low-quality or irrelevant query engine responses.

Source File: llama-index-core/llama_index/core/tools/eval_query_engine.py

Module: llama_index.core.tools.eval_query_engine

Lines of Code: 97

Dependencies

Dependency	Type	Purpose
`llama_index.core.base.base_query_engine.BaseQueryEngine`	Internal	Base class for query engines
`llama_index.core.evaluation.AnswerRelevancyEvaluator`	Internal	Default evaluator for answer relevancy
`llama_index.core.evaluation.BaseEvaluator`	Internal	Abstract base class for evaluators
`llama_index.core.evaluation.EvaluationResult`	Internal	Data class holding evaluation results
`llama_index.core.tools.QueryEngineTool`	Internal	Parent class providing query engine tool functionality
`llama_index.core.tools.types.ToolMetadata`	Internal	Metadata for tool registration
`llama_index.core.tools.types.ToolOutput`	Internal	Standard tool output wrapper

Module Constants

Constant	Value	Description
`DEFAULT_NAME`	`"query_engine_tool"`	Default tool name
`DEFAULT_DESCRIPTION`	`"Useful for running a natural language query against a knowledge base..."`	Default tool description
`FAILED_TOOL_OUTPUT_TEMPLATE`	`"Could not use tool {tool_name} because it failed evaluation.\nReason: {reason}"`	Template for failed evaluation output

Class: EvalQueryEngineTool

class EvalQueryEngineTool(QueryEngineTool)

Inheritance

EvalQueryEngineTool inherits from QueryEngineTool, which itself inherits from BaseTool. This allows the evaluated tool to be used anywhere a standard query engine tool is accepted.

Private Attributes

Attribute	Type	Description
`_evaluator`	`BaseEvaluator`	The evaluator instance used to assess query engine responses
`_failed_tool_output_template`	`str`	Template string for generating failure messages

Constructor

def __init__(
    self,
    evaluator: BaseEvaluator,
    *args: Any,
    failed_tool_output_template: str = FAILED_TOOL_OUTPUT_TEMPLATE,
    **kwargs: Any,
)

The constructor accepts an evaluator and an optional failed_tool_output_template. All other arguments are passed through to the parent QueryEngineTool.__init__.

Method: _process_tool_output

def _process_tool_output(
    self,
    tool_output: ToolOutput,
    evaluation_result: EvaluationResult,
) -> ToolOutput

This private method processes the tool output based on the evaluation result:

If evaluation_result.passing is True, the original tool_output is returned unchanged.
If the evaluation fails, the tool output's content is replaced with the failure template, formatted with the tool name and the evaluation feedback.

def _process_tool_output(self, tool_output, evaluation_result):
    if evaluation_result.passing:
        return tool_output
    tool_output.content = self._failed_tool_output_template.format(
        tool_name=self.metadata.name,
        reason=evaluation_result.feedback,
    )
    return tool_output

Class Method: from_defaults

@classmethod
def from_defaults(
    cls,
    query_engine: BaseQueryEngine,
    name: Optional[str] = None,
    description: Optional[str] = None,
    return_direct: bool = False,
    resolve_input_errors: bool = True,
    evaluator: Optional[BaseEvaluator] = None,
) -> "EvalQueryEngineTool"

Factory method that creates an EvalQueryEngineTool with sensible defaults:

If no evaluator is provided, an AnswerRelevancyEvaluator() is used.
If no name is provided, DEFAULT_NAME ("query_engine_tool") is used.
If no description is provided, DEFAULT_DESCRIPTION is used.

Method: call (Synchronous)

def call(self, *args: Any, **kwargs: Any) -> ToolOutput

Invokes the parent QueryEngineTool.call to get the raw tool output.
Runs the evaluator's evaluate_response method with the original input and the raw output.
Passes both the tool output and evaluation result to _process_tool_output.
Returns the potentially modified tool output.

Method: acall (Asynchronous)

async def acall(self, *args: Any, **kwargs: Any) -> ToolOutput

The async counterpart:

Invokes the parent QueryEngineTool.acall with await.
Runs the evaluator's aevaluate_response method with await.
Passes both results to _process_tool_output.

Execution Flow

Agent sends query
    |
    v
EvalQueryEngineTool.call(query)
    |
    v
QueryEngineTool.call(query)  -- parent class
    |
    v
Query Engine produces response
    |
    v
Evaluator.evaluate_response(query, response)
    |
    +---> passing=True  --> return original ToolOutput
    |
    +---> passing=False --> replace content with failure message, return modified ToolOutput

Design Patterns

Decorator Pattern

EvalQueryEngineTool follows the decorator pattern: it wraps an existing QueryEngineTool and adds evaluation behavior without modifying the parent class. The tool can be used as a drop-in replacement for QueryEngineTool anywhere in the agent pipeline.

Fail-Safe Output

When evaluation fails, the tool does not raise an exception. Instead, it replaces the output content with a descriptive failure message. This allows the agent to receive structured feedback about why a tool's response was unsatisfactory and potentially retry with a different approach.

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment