Principle:Explodinggradients Ragas Framework Message Conversion

Framework Message Conversion

Framework Message Conversion is the principle of translating framework-specific message representations into a standard set of Ragas message types, enabling framework-agnostic evaluation of multi-turn LLM agent conversations.

Theoretical Foundation

The Problem of Divergent Message Formats

LLM application frameworks (LangChain/LangGraph, OpenAI Swarm, LlamaIndex, and others) each define their own message types to represent conversations between users, AI agents, and tools. These representations differ in class hierarchies, field names, metadata handling, and how tool calls are structured.

For evaluation to be framework-agnostic, a common message representation is needed. Without it, every evaluation metric would need to understand every framework's message format, creating an unsustainable combinatorial explosion of adapters.

Standard Message Types

Ragas defines three standard message types that capture the essential information from any framework:

• HumanMessage: Represents user input. Contains a content string and optional metadata.
• AIMessage: Represents AI agent responses. Contains a content string, optional tool_calls (a list of ToolCall objects), and optional metadata.
• ToolMessage: Represents tool/function execution results. Contains a content string and optional metadata.

Each ToolCall has a name (the function name) and args (a dictionary of arguments).

Conversion Principles

Framework message conversion follows several key principles:

Role-Based Dispatching

Messages are classified by their role (human/user, AI/assistant, tool) and dispatched to the appropriate Ragas message constructor. Framework-specific role indicators (class types, string role fields) are mapped to the standard types.

Tool Call Preservation

Tool calls embedded in AI messages must be extracted and converted to standard ToolCall objects. Different frameworks encode tool calls differently:

• Some use nested dictionaries with function.name and function.arguments keys
• Some store arguments as JSON strings that need parsing
• Some use dedicated tool call objects with typed fields

The conversion must normalize all these representations into the standard ToolCall(name, args) format.

Metadata Handling

Framework messages may carry metadata (response IDs, timestamps, model names, token counts). Conversion can optionally preserve this metadata in the Ragas message's metadata field, which is useful for debugging but not required for evaluation.

Graceful Handling of System Messages

Some frameworks include system messages (system prompts) in the conversation. Since these are not part of the user-agent interaction being evaluated, they are typically skipped during conversion.

Enabling Framework-Agnostic Evaluation

Once messages are converted to Ragas standard types, any evaluation metric (tool call accuracy, goal accuracy, topic adherence, etc.) can operate on the conversation without knowledge of the original framework. This separation of concerns allows:

• New metrics to be added without framework-specific code
• New framework integrations to be added without modifying existing metrics
• Consistent evaluation results regardless of the underlying framework

Implemented By

• LangGraph Convert Messages -- conversion from LangChain/LangGraph message types to Ragas messages
• Swarm Convert Messages -- conversion from OpenAI Swarm dictionary messages to Ragas messages

See Also

• Implementation:Explodinggradients_Ragas_LangGraph_Convert_Messages
• Multi-Turn Evaluation Schema -- the schema that uses the converted messages for evaluation
• Tool Call Accuracy Evaluation -- a metric that operates on the converted messages
• Agent Goal Accuracy Evaluation -- another metric that operates on the converted messages