Implementation:Explodinggradients Ragas Sample Annotated Summary Schema

Field	Value
source	Explodinggradients_Ragas|https://github.com/explodinggradients/ragas
domains	Data, Evaluation
last_updated	2026-02-10 00:00 GMT

Overview

A JSON data fixture containing annotated evaluation samples for the summary_accuracy metric, providing user-provided text, generated summaries, LLM verdicts, and human acceptance judgments.

Description

The sample_annotated_summary.json file is a static data fixture in the documentation assets directory. It contains an array of annotated evaluation samples keyed under "summary_accuracy". Each sample pairs a source text (provided via user_input) with a generated summary (in response). The metric evaluates whether the summary accurately captures the key facts from the source text. The single_turn_aspect_critic_prompt records the LLM's verdict with a reason explaining whether the summary faithfully represents the original content. Unlike the answer_correctness fixture, this data does not include a separate reference field since the source text itself serves as the ground truth. The is_accepted flag indicates human reviewer agreement with the overall annotation.

Usage

This file is consumed by the Ragas documentation site to illustrate the metric alignment workflow for summarization tasks. It can be loaded programmatically for testing or calibrating summarization evaluation pipelines.

import json

with open("docs/_static/sample_annotated_summary.json", "r") as f:
    data = json.load(f)

summary_samples = data["summary_accuracy"]
print(f"Total samples: {len(summary_samples)}")

Code Reference

Field	Value
Source Location	docs/_static/sample_annotated_summary.json
Structure	Top-level JSON object with a single key "summary_accuracy" mapping to an array of annotation objects
File Size	460 lines

Data Schema

Field	Type	Description
summary_accuracy	Array[Object]	Array of annotation samples for the summary_accuracy metric
summary_accuracy[].metric_input	Object	Contains user_input (string, source text to summarize) and response (string, generated summary)
summary_accuracy[].metric_output	Integer	Binary score: 1 (accurate summary) or 0 (inaccurate summary)
summary_accuracy[].prompts	Object	Contains single_turn_aspect_critic_prompt with nested evaluation trace
summary_accuracy[].prompts.*.prompt_input	Object	Full context: user_input, response, retrieved_contexts (null), reference_contexts (null), reference (null)
summary_accuracy[].prompts.*.prompt_output	Object	LLM judgment: reason (string) and verdict (integer)
summary_accuracy[].prompts.*.edited_output	null	Human-edited judgment or null if unedited
summary_accuracy[].is_accepted	Boolean	Whether the human reviewer accepted the annotation

Usage Examples

import json

# Load the summary annotation data
with open("docs/_static/sample_annotated_summary.json", "r") as f:
    data = json.load(f)

samples = data["summary_accuracy"]

# Compute accuracy rate
accurate = sum(1 for s in samples if s["metric_output"] == 1)
print(f"Accurate summaries: {accurate}/{len(samples)} ({accurate/len(samples):.0%})")

# Show accepted vs rejected annotations
accepted = [s for s in samples if s["is_accepted"]]
rejected = [s for s in samples if not s["is_accepted"]]
print(f"Accepted: {len(accepted)}, Rejected: {len(rejected)}")

# Inspect a failed summary
for s in samples:
    if s["metric_output"] == 0:
        reason = s["prompts"]["single_turn_aspect_critic_prompt"]["prompt_output"]["reason"]
        print(f"Failure reason: {reason}")
        break

Related Pages

• Implementation:Explodinggradients_Ragas_Annotated_Data_Schema -- Similar annotation fixture for helpfulness metric
• Implementation:Explodinggradients_Ragas_Edited_Chain_Runs_Schema -- Similar annotation fixture for answer_correctness metric
• Implementation:Explodinggradients_Ragas_MkDocs_Configuration -- Documentation site configuration that serves this data