Implementation:Explodinggradients Ragas Annotated Data Schema

Field	Value
source	Explodinggradients_Ragas|https://github.com/explodinggradients/ragas
domains	Data, Evaluation
last_updated	2026-02-10 00:00 GMT

Overview

A JSON data fixture containing sample annotation data for the helpfulness metric, providing metric inputs, outputs, prompt traces, and human acceptance judgments used for metric alignment and training.

Description

The annotated_data.json file is a static data fixture stored in the documentation assets directory. It contains an array of annotated evaluation samples keyed under the "helpfulness" metric name. Each sample records a complete evaluation trace: the original user input and LLM response (metric_input), the numeric metric score (metric_output), the full prompt chain including both the LLM-generated output and optional human-edited output (prompts), and a boolean indicating whether the annotation was accepted by a human reviewer (is_accepted). The file serves as reference data for the "Train and Align Metrics" documentation workflow, demonstrating how human annotations can be collected and used to calibrate the AspectCritic-based helpfulness metric.

Usage

This file is consumed by the Ragas documentation site to illustrate the metric alignment workflow. It is loaded via JavaScript in the docs pages and referenced in the "Train your own metric" how-to guide. It can also be parsed programmatically for testing metric alignment pipelines.

import json

with open("docs/_static/annotated_data.json", "r") as f:
    data = json.load(f)

helpfulness_samples = data["helpfulness"]
for sample in helpfulness_samples:
    print(sample["metric_output"], sample["is_accepted"])

Code Reference

Field	Value
Source Location	docs/_static/annotated_data.json
Structure	Top-level JSON object with a single key "helpfulness" mapping to an array of annotation objects
File Size	532 lines

Data Schema

Field	Type	Description
helpfulness	Array[Object]	Array of annotation samples for the helpfulness metric
helpfulness[].metric_input	Object	Contains user_input (string) and response (string)
helpfulness[].metric_output	Integer	Binary score: 1 (helpful) or 0 (not helpful)
helpfulness[].prompts	Object	Contains single_turn_aspect_critic_prompt with nested prompt_input, prompt_output, and optional edited_output
helpfulness[].prompts.*.prompt_input	Object	Full prompt context: user_input, response, retrieved_contexts, reference_contexts, reference
helpfulness[].prompts.*.prompt_output	Object	LLM judgment: reason (string) and verdict (integer)
helpfulness[].prompts.*.edited_output	null	Human-edited judgment: reason (string) and verdict (integer), or null if unedited
helpfulness[].is_accepted	Boolean	Whether the human reviewer accepted the annotation

Usage Examples

import json

# Load the annotated data
with open("docs/_static/annotated_data.json", "r") as f:
    data = json.load(f)

# Count accepted vs rejected annotations
samples = data["helpfulness"]
accepted = sum(1 for s in samples if s["is_accepted"])
rejected = len(samples) - accepted
print(f"Accepted: {accepted}, Rejected: {rejected}")

# Find samples where human edited the output
edited = [s for s in samples
          if s["prompts"]["single_turn_aspect_critic_prompt"]["edited_output"] is not None]
print(f"Human-edited samples: {len(edited)}")

Related Pages

• Explodinggradients_Ragas_Edited_Chain_Runs_Schema -- Similar annotation fixture for answer_correctness metric
• Explodinggradients_Ragas_Sample_Annotated_Summary_Schema -- Similar annotation fixture for summary_accuracy metric
• Explodinggradients_Ragas_MkDocs_Configuration -- Documentation site configuration that serves this data