Principle:Explodinggradients Ragas Tool Call Accuracy Evaluation

Tool Call Accuracy Evaluation

Tool Call Accuracy Evaluation is the principle of measuring how correctly an AI agent selects and invokes tools during multi-turn conversations. When autonomous agents interact with external tools, three dimensions of correctness must be assessed: whether the agent chose the right tool, whether it passed the right arguments, and whether it invoked tools in the right order. Tool call accuracy evaluation provides a unified score that captures all three dimensions.

Theoretical Foundation

Tool Selection and Argument Matching

In agentic systems, an AI agent is given access to a set of tools (functions, APIs, or external services) and must decide which tool to call and what arguments to pass based on the user's request. Evaluating tool call accuracy requires comparing the agent's predicted tool calls against a set of reference (ground-truth) tool calls.

The evaluation decomposes into two orthogonal aspects:

Sequence alignment: Whether the predicted tool call names match the reference tool call names in the required order. This determines a binary gate -- if the sequence does not align, the overall score is zero.
Argument accuracy: For each pair of aligned tool calls, how well the predicted arguments match the reference arguments. This is computed per-argument using a configurable comparison metric (defaulting to exact string match) and averaged across all reference arguments.

Strict vs. Flexible Ordering

Two evaluation modes address different agent architectures:

Strict order mode: The predicted tool call sequence must exactly match the reference sequence. This is appropriate when tool calls have dependencies and must execute sequentially (e.g., search before booking).
Flexible order mode: The predicted and reference tool call sequences are sorted before comparison, allowing any permutation. This accommodates parallel or order-independent tool execution.

Score Computation

The final score is computed as:

score = sequence_alignment_factor * average_argument_accuracy * coverage_penalty

Where:

sequence_alignment_factor is 1 if the tool call name sequences align (under the chosen ordering mode), 0 otherwise
average_argument_accuracy is the mean of per-tool-call argument scores, where each argument score is computed by the configured comparison metric
coverage_penalty is applied when the predicted and reference tool call lists differ in length, calculated as min(predicted_count, reference_count) / reference_count

This multiplicative structure ensures that a wrong tool sequence immediately yields zero, while partial argument matches produce proportional scores.

Edge Cases

The evaluation handles several boundary conditions:

Both predicted and reference tool calls are empty: returns 1.0 (trivially correct)
No predicted tool calls but reference exists: returns 0.0
Length mismatch: only the overlapping prefix is compared, with a proportional coverage penalty applied

Relationship to Other Concepts

Tool call accuracy evaluates individual tool call correctness in an ordered or unordered fashion. It complements Tool Call F1 Evaluation, which takes a set-based precision/recall approach without considering order. Together, these metrics provide a comprehensive view of agent tool-use behavior.

The evaluation operates on multi-turn conversation samples that contain both the actual conversation messages and reference tool calls for comparison.

Implemented By

ToolCallAccuracy Metric -- the Ragas metric class that implements this evaluation principle

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment