Principle:Explodinggradients Ragas Test Query Synthesis
| Knowledge Sources | Domains | Last Updated |
|---|---|---|
| explodinggradients/ragas | LLM Evaluation, Test Data Generation, Query Synthesis | 2026-02-10 |
Overview
Description
Test Query Synthesis is the principle of generating diverse, realistic test queries from knowledge graphs using scenario-based generation and persona-driven perspectives. The core idea is that effective evaluation of LLM applications requires test sets that cover different question types (single-hop factual, multi-hop reasoning, abstract conceptual), different user perspectives (personas), and different areas of the source content. Ragas achieves this by defining a query distribution -- a weighted mixture of query synthesizers -- that controls the balance and diversity of the generated test set.
Usage
Test query synthesis is the culmination of the Ragas test generation pipeline. After documents have been loaded, the knowledge graph constructed and enriched, and personas generated, the TestsetGenerator.generate() method orchestrates the synthesis process:
- The query distribution is resolved: either a user-provided distribution or the default distribution consisting of
SingleHopSpecificQuerySynthesizer,MultiHopAbstractQuerySynthesizer, andMultiHopSpecificQuerySynthesizer, each with equal probability. - The target testset_size is split proportionally across the synthesizers based on their probabilities.
- Each synthesizer generates scenarios from the knowledge graph -- each scenario defines a context (a set of nodes) and a persona for the question.
- Each scenario is then converted to a sample by the synthesizer, which uses the LLM to produce a question, reference answer, and associated metadata.
- The results are collected into a
Testsetobject.
Theoretical Basis
Scenario-Based Generation: Rather than generating queries directly from raw text, Ragas first constructs scenarios from the knowledge graph. A scenario encapsulates a specific context (one or more related nodes), a persona, and metadata about the query type. This two-phase approach (scenario generation then sample generation) separates the concerns of what to ask about from how to ask it, enabling more structured and controllable generation.
Query Type Diversity: The default query distribution includes three synthesizer types:
- SingleHopSpecificQuerySynthesizer: Generates factual questions answerable from a single node's content. These test basic retrieval and comprehension.
- MultiHopAbstractQuerySynthesizer: Generates conceptual questions requiring synthesis across multiple related nodes. These test the ability to form abstract understanding across sources.
- MultiHopSpecificQuerySynthesizer: Generates specific factual questions requiring information from multiple nodes. These test multi-step retrieval and reasoning.
Persona-Driven Perspective Diversity: Each scenario is assigned a persona from the generated persona list. The persona influences the framing, vocabulary, and focus of the generated question. By cycling through personas across scenarios, the system ensures that the test set represents diverse user perspectives rather than a uniform question style.
Proportional Split with Distribution Control: The query_distribution parameter allows fine-grained control over the composition of the test set. Each entry is a tuple of (synthesizer, probability). The system uses calculate_split_values() to convert probabilities into integer counts that sum to testset_size, handling rounding appropriately.
Adaptive Distribution: When using the default distribution, the system checks which synthesizers are compatible with the current knowledge graph (i.e., whether get_node_clusters(kg) returns results). Incompatible synthesizers are automatically excluded, and probabilities are redistributed equally among the remaining ones. This prevents errors when the graph lacks the structure needed for certain query types.
Parallel Execution: Both scenario generation and sample generation use an Executor for parallel async execution, enabling efficient generation even for large test sets. The executor supports configurable batch sizes, run configurations, and callback-based monitoring.
Related Pages
- Implementation:Explodinggradients_Ragas_TestsetGenerator_Generate
- Principle:Explodinggradients_Ragas_Knowledge_Graph_Construction -- the graph from which queries are synthesized
- Principle:Explodinggradients_Ragas_Knowledge_Graph_Enrichment -- enrichment that enables multi-hop and abstract queries
- Principle:Explodinggradients_Ragas_Persona_Generation -- personas that diversify query perspectives
- Principle:Explodinggradients_Ragas_Testset_Export -- exporting the synthesized test set to usable formats