Workflow:Explodinggradients Ragas Test Data Generation
| Knowledge Sources | |
|---|---|
| Domains | LLMs, RAG, Test_Generation, Data_Engineering |
| Last Updated | 2026-02-10 06:00 GMT |
Overview
End-to-end process for generating synthetic test datasets from source documents using Ragas knowledge graphs, persona generation, and query synthesizers.
Description
This workflow covers the automated generation of evaluation test datasets for RAG systems. It starts from raw source documents, builds a knowledge graph with nodes (documents, chunks) and relationships, enriches the graph with extracted entities, summaries, and embeddings, generates user personas from document clusters, and synthesizes diverse test queries (single-hop and multi-hop) with reference contexts and answers. The output is a structured Testset ready for use in RAG evaluation experiments.
Usage
Execute this workflow when you need to create a comprehensive test dataset for evaluating a RAG system but do not have manually curated test questions. You should have a collection of source documents (text, markdown, PDFs) that represent your knowledge base. This workflow is particularly valuable when you need diverse query types (factual, interpretive, multi-hop reasoning) and want test data that reflects realistic user interactions.
Execution Steps
Step 1: Load Source Documents
Load the raw source documents that form the knowledge base for your RAG system. Documents can be loaded from various sources using LangChain document loaders, LlamaIndex readers, or custom file parsers. Each document should contain the text content and optional metadata (source URL, title, date).
Key considerations:
- Supports multiple document formats (text, markdown, PDF, HTML)
- Documents can be pre-chunked or split automatically by the generator
- For pre-chunked data, use the generate_with_chunks method to skip internal splitting
- Ensure documents have sufficient content for meaningful query generation
Step 2: Build the Knowledge Graph
Transform the loaded documents into a knowledge graph. The graph represents documents and chunks as nodes, with relationships capturing semantic connections. The TestsetGenerator automatically handles document chunking, node creation, and initial relationship building.
Key considerations:
- Documents are split into chunks and represented as DOCUMENT and CHUNK node types
- Parent-child relationships link documents to their chunks
- The graph structure enables both single-hop and multi-hop query generation
- Custom chunk sizes and overlap can be configured
Step 3: Enrich the Knowledge Graph
Apply transformation pipelines to enrich the graph with extracted information. Extractors pull entities, keyphrases, and summaries from each node. Relationship builders establish connections between nodes based on entity overlap (Jaccard similarity) or embedding similarity (cosine similarity). Multiple extractors and builders run in parallel for efficiency.
Key considerations:
- Extractors add properties like summaries, entities, and embeddings to nodes
- Cosine similarity builders use embedding vectors with configurable thresholds (default 0.75)
- Jaccard similarity builders use entity overlap for relationship creation
- The Leiden clustering algorithm identifies groups of related nodes for multi-hop scenarios
Step 4: Generate User Personas
Create diverse user personas from the knowledge graph to produce varied query styles. The PersonaGenerator clusters similar document summaries using embedding similarity, then uses an LLM to generate representative personas for each cluster. Personas influence the tone, complexity, and focus of generated queries.
Key considerations:
- Personas are generated from document clusters, not individual documents
- Each persona has a name and role description (e.g., "Data Scientist researching NLP")
- The number and diversity of personas depends on the document collection
- Personas ensure test queries cover different user perspectives
Step 5: Synthesize Test Queries
Generate test queries using configurable query synthesizers. The synthesizers combine graph paths, personas, and query templates to produce diverse questions. Single-hop synthesizers create questions answerable from a single chunk, while multi-hop synthesizers create questions requiring information from multiple nodes connected via graph relationships.
Key considerations:
- Default distribution: 50% single-hop specific, 25% multi-hop abstract, 25% multi-hop specific
- Query styles include web search, conversational, and factual formats
- Query lengths can be short, medium, or long
- Each query includes reference contexts and expected answers
- Custom synthesizers can be created by subclassing SingleHopQuerySynthesizer or MultiHopQuerySynthesizer
Step 6: Export the Testset
Export the generated testset for use in evaluation experiments. The Testset can be converted to a pandas DataFrame, saved as CSV/JSONL, or directly used with the Ragas evaluation framework. Each sample contains the generated query, reference contexts, and reference answer.
Key considerations:
- Export to pandas DataFrame via testset.to_pandas()
- The knowledge graph can be saved and loaded for reuse via kg.save() and kg.load()
- Testset samples include metadata about query type, persona, and source nodes
- The testset can be used directly with the @experiment decorator for evaluation