Overview
A SQL dataset validation pipeline that loads a CSV dataset, executes each SQL query against the BookSQL SQLite database, captures results and errors, and produces detailed validation reports with summary statistics.
Description
The validate_sql_dataset.py module validates Text-to-SQL datasets by executing every SQL query against the target database and classifying each result. The load_dataset function reads a CSV file with columns Query, SQL, Levels, and split, and returns a list of dictionaries. The execute_and_validate_query function executes a single query using execute_sql from db_utils, records execution time, captures result shape and columns, classifies the result type (has_data, null_values, empty, or failed), and truncates results to 10 rows for manageable output. The generate_summary_statistics function aggregates results into a summary with overall success rates, per-difficulty-level breakdowns, result type distributions, common error types, and average execution times. The main function orchestrates the full pipeline and writes validation_results.json and validation_summary.json output files.
Usage
# Run the full validation pipeline
python -m ragas_examples.text2sql.validate_sql_dataset
Code Reference
| Field |
Value
|
| Source Location |
examples/ragas_examples/text2sql/validate_sql_dataset.py
|
| File Size |
317 lines
|
| Import |
from ragas_examples.text2sql.validate_sql_dataset import load_dataset, execute_and_validate_query, generate_summary_statistics
|
Function Signatures
def load_dataset(csv_path: str = "datasets/booksql_sample.csv") -> List[Dict[str, Any]]
def execute_and_validate_query(query_data: Dict[str, Any]) -> Dict[str, Any]
def generate_summary_statistics(results: List[Dict[str, Any]]) -> Dict[str, Any]
I/O Contract
| Function |
Input |
Output
|
| load_dataset |
csv_path: str (path to CSV with Query, SQL, Levels, split columns) |
List[Dict] with keys: index, query, sql, level, split
|
| execute_and_validate_query |
query_data: Dict with index, query, sql, level, split |
Dict with execution_success, execution_time, error_message, result_data, result_shape, result_columns, result_type, result_truncated, total_rows
|
| generate_summary_statistics |
results: List[Dict] (output of execute_and_validate_query) |
Dict with total_queries, successful_queries, failed_queries, overall_success_rate, average_execution_time_seconds, result_type_counts, statistics_by_difficulty, common_error_types
|
Result Type Classification
| Result Type |
Condition
|
has_data |
Query succeeded and returned at least one row with non-null values
|
null_values |
Query succeeded but all values in the first row are null
|
empty |
Query succeeded but returned zero rows
|
failed |
Query execution raised an error
|
Usage Examples
from ragas_examples.text2sql.validate_sql_dataset import (
load_dataset,
execute_and_validate_query,
generate_summary_statistics,
)
# Load the dataset
dataset = load_dataset("datasets/booksql_sample.csv")
# Validate all queries
results = []
for query_data in dataset:
result = execute_and_validate_query(query_data)
results.append(result)
# Generate summary
summary = generate_summary_statistics(results)
print(f"Success rate: {summary['overall_success_rate']:.1%}")
print(f"Queries with data: {summary['result_type_counts']['has_data']}")
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
