Principle:Explodinggradients Ragas Multi Turn Evaluation Schema

Multi-Turn Evaluation Schema

Multi-Turn Evaluation Schema is the principle of using a structured data model to represent multi-turn agent conversations along with their associated reference data for evaluation purposes. This schema provides the standard input format consumed by all multi-turn evaluation metrics in Ragas.

Theoretical Foundation

The Need for Structured Conversation Representation

Evaluating multi-turn AI agent interactions requires more than just a list of messages. The evaluation system needs:

Typed messages: Each message must be classified as human input, AI response, or tool output, with appropriate fields for each type (e.g., tool calls on AI messages).
Reference data: Ground-truth information for comparison, such as expected tool calls, desired outcomes, or allowed topics.
Validation rules: Structural constraints that ensure the conversation is well-formed (e.g., tool messages must follow AI messages that made tool calls).

A formal schema encodes all these requirements, enabling consistent data validation and predictable metric behavior.

Message Type System

The schema builds on three typed message classes:

HumanMessage: Represents user input with a content string and optional metadata.
AIMessage: Represents AI agent responses with a content string, optional tool_calls (list of ToolCall objects, each with a name and args dictionary), and optional metadata.
ToolMessage: Represents tool execution results with a content string and optional metadata.

This type system enforces a clear distinction between the roles in an agent conversation and ensures that tool call metadata is preserved where needed.

Conversation Validation Rules

The schema enforces structural invariants on the message sequence:

ToolMessage must be preceded by an AIMessage: A tool response can only appear after an AI message has appeared somewhere earlier in the conversation, establishing that tools are invoked by the AI.
ToolMessage must immediately follow an AIMessage or another ToolMessage: Tool responses form contiguous blocks following the AI message that invoked them.
AIMessage preceding a ToolMessage must have tool_calls: The AI message that directly precedes a tool response block must have invoked at least one tool.

These rules ensure that the conversation structure is physically plausible -- tools do not respond unless called, and tool responses are grouped with their invoking message.

Reference Data Fields

The schema supports multiple optional reference fields, each enabling a different category of evaluation metric:

Field	Type	Enables
`reference`	`Optional[str]`	Goal accuracy evaluation (with reference), general expected outcome comparison
`reference_tool_calls`	`Optional[List[ToolCall]]`	Tool call accuracy and F1 evaluation
`reference_topics`	`Optional[List[str]]`	Topic adherence evaluation
`rubrics`	`Optional[Dict[str, str]]`	Rubric-based evaluation

This design allows a single sample to carry reference data for multiple metrics simultaneously, enabling batch evaluation across different dimensions.

Pretty Representation

The schema provides a human-readable serialization of the conversation (via pretty_repr()) that formats each message with role labels and indentation. This representation is used by LLM-based metrics (such as goal accuracy and topic adherence) when passing the conversation to an evaluator LLM for analysis.

Relationship to Other Concepts

The multi-turn evaluation schema is the foundational data structure consumed by all agent evaluation metrics:

Tool Call Accuracy Evaluation uses user_input and reference_tool_calls
Tool Call F1 Evaluation uses user_input and reference_tool_calls
Agent Goal Accuracy Evaluation uses user_input and optionally reference
Topic Adherence Evaluation uses user_input and reference_topics

Framework integration modules (Framework Message Conversion, Swarm Message Conversion) produce the typed messages that populate the user_input field.

Implemented By

MultiTurnSample Class -- the Pydantic model that implements this schema

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment