Principle:Explodinggradients Ragas Agent Goal Accuracy Evaluation

Agent Goal Accuracy Evaluation

Agent Goal Accuracy Evaluation is the principle of assessing whether an AI agent successfully achieved its intended goal during a multi-turn conversation. Rather than examining individual tool calls or responses, goal accuracy takes a holistic view of the entire conversation trajectory and determines whether the desired outcome was reached.

Theoretical Foundation

Goal-Oriented Agent Assessment

Autonomous AI agents are designed to accomplish user-specified tasks through sequences of actions including tool calls, information retrieval, and conversational turns. While lower-level metrics evaluate individual actions (e.g., whether the right tool was called), goal accuracy evaluates the final outcome of the entire interaction: did the agent accomplish what the user asked?

This top-level evaluation is critical because an agent may make imperfect intermediate decisions yet still achieve the desired outcome, or conversely, make correct individual actions but fail to reach the overall goal due to logical errors in the workflow.

Two Evaluation Modes

Goal accuracy evaluation supports two distinct modes, addressing different evaluation scenarios:

Reference-Based Evaluation

In this mode, a human-provided reference outcome is compared against the agent's actual outcome. The evaluation follows a two-step LLM-based process:

1. Outcome inference: An LLM analyzes the full conversation transcript and extracts the user's goal and the end state (what actually happened)
2. Outcome comparison: The extracted end state is compared against the reference outcome using a second LLM prompt that determines whether they are equivalent

This mode is appropriate when ground-truth expected outcomes are available, such as in benchmark datasets or regression test suites.

Reference-Free Evaluation

In this mode, no external reference is provided. The evaluation follows the same two-step process, but with a key difference:

1. Outcome inference: An LLM analyzes the full conversation transcript and extracts both the user's goal and the end state
2. Outcome comparison: The inferred user goal is compared against the inferred end state to determine whether the agent fulfilled what the user originally asked for

This mode is appropriate for production monitoring scenarios where expected outcomes are not known in advance, or when evaluating open-ended agent interactions.

LLM-as-Judge Pattern

Both modes rely on the LLM-as-judge pattern, where a separate LLM evaluates the agent's performance. The evaluation uses structured prompts with Pydantic models to ensure consistent output parsing:

• InferGoalOutcomePrompt: Extracts user_goal and end_state from the conversation workflow
• CompareOutcomePrompt: Compares a desired outcome against an achieved outcome and returns a binary verdict ("0" or "1")

The evaluation output is binary -- the agent either achieved the goal (1.0) or did not (0.0).

Conversation Representation

The full multi-turn conversation is serialized into a human-readable text format (using pretty_repr()) that preserves the structure of Human messages, AI responses, tool calls, and tool outputs. This formatted representation is passed to the LLM for analysis, ensuring that the evaluator has full context about the agent's behavior.

Relationship to Other Concepts

Goal accuracy operates at a higher abstraction level than tool-level metrics. While Tool Call Accuracy Evaluation and Tool Call F1 Evaluation assess individual tool calls, goal accuracy evaluates the end result of the entire workflow.

Goal accuracy complements Topic Adherence Evaluation, which checks whether the agent stayed within designated topic boundaries. An agent could achieve its goal while straying off-topic, or stay on-topic while failing to complete the task.

The evaluation operates on multi-turn conversation samples and requires an LLM for judgment.

Implemented By

• AgentGoalAccuracy Metric -- the Ragas metric classes (AgentGoalAccuracyWithReference and AgentGoalAccuracyWithoutReference) that implement this evaluation principle

See Also

• Implementation:Explodinggradients_Ragas_AgentGoalAccuracy_Metric
• Tool Call Accuracy Evaluation -- evaluating individual tool call correctness
• Topic Adherence Evaluation -- evaluating whether the agent stays on topic
• Multi-Turn Evaluation Schema -- the data schema for multi-turn samples