Implementation:Explodinggradients Ragas MetricAnnotation Class

MetricAnnotation Class

MetricAnnotation, SingleMetricAnnotation, and SampleAnnotation implement the Human Annotation Collection principle in the Ragas evaluation toolkit. They provide the structured data format for human-annotated evaluation data used during prompt optimization.

Source Location

File: src/ragas/dataset_schema.py
PromptAnnotation: Lines 555-561
SampleAnnotation: Lines 564-572
MetricAnnotation: Lines 575-618
SingleMetricAnnotation: Lines 621-838

Import

from ragas.dataset_schema import MetricAnnotation, SingleMetricAnnotation

Additional imports for building annotations programmatically:

from ragas.dataset_schema import SampleAnnotation, PromptAnnotation

Class Hierarchy

BaseModel (Pydantic)
  ├── PromptAnnotation
  ├── SampleAnnotation
  ├── MetricAnnotation
  └── SingleMetricAnnotation

All classes inherit from Pydantic's BaseModel.

PromptAnnotation (Lines 555-561)

class PromptAnnotation(BaseModel):
    prompt_input: Dict[str, Any]
    prompt_output: Dict[str, Any]
    edited_output: Optional[Dict[str, Any]] = None

Captures the input/output behavior of a single prompt within a metric, along with an optional human-corrected output.

Field	Type	Default	Description
`prompt_input`	`Dict[str, Any]`	required	The input dictionary passed to the prompt.
`prompt_output`	`Dict[str, Any]`	required	The original output produced by the prompt.
`edited_output`	`Optional[Dict[str, Any]]`	`None`	Human-corrected output if the original was incorrect.

SampleAnnotation (Lines 564-572)

class SampleAnnotation(BaseModel):
    metric_input: Dict[str, Any]
    metric_output: float
    prompts: Dict[str, PromptAnnotation]
    is_accepted: bool
    target: Optional[float] = None

Represents a single annotated sample with the metric-level input/output and prompt-level details.

Field	Type	Default	Description
`metric_input`	`Dict[str, Any]`	required	Input fields for the metric (e.g., user_input, response, reference).
`metric_output`	`float`	required	The metric's original predicted output value.
`prompts`	`Dict[str, PromptAnnotation]`	required	Prompt-level annotations keyed by prompt name.
`is_accepted`	`bool`	required	Whether the human accepted the metric's output.
`target`	`Optional[float]`	`None`	Optional human-provided target score.

Supports item access via sample["field_name"] through a __getitem__ method.

MetricAnnotation (Lines 575-618)

class MetricAnnotation(BaseModel):
    root: Dict[str, List[SampleAnnotation]]

Top-level container that holds annotations for multiple metrics, keyed by metric name.

from_json() (Lines 611-615)

@classmethod
def from_json(cls, path: str, metric_name: Optional[str]) -> MetricAnnotation

Loads annotations from a JSON file. If metric_name is provided, only that metric's annotations are loaded.

Parameters:

Parameter	Type	Description
`path`	`str`	Path to the JSON file containing annotations.
`metric_name`	`Optional[str]`	If specified, filter to only this metric. Raises `ValueError` if not found.

Subscript Access

annotation = MetricAnnotation.from_json("annotations.json", metric_name=None)
single = annotation["my_metric"]  # Returns SingleMetricAnnotation

The __getitem__ method (line 578-579) creates a SingleMetricAnnotation from the stored data.

len (Line 617-618)

Returns the total number of annotations across all metrics.

SingleMetricAnnotation (Lines 621-838)

class SingleMetricAnnotation(BaseModel):
    name: str
    samples: List[SampleAnnotation]

The primary data structure consumed by optimizers. Contains all annotations for a single metric.

Field	Type	Description
`name`	`str`	Name of the metric.
`samples`	`List[SampleAnnotation]`	List of annotated samples.

Core Methods

to_evaluation_dataset() (Lines 625-627)

def to_evaluation_dataset(self) -> EvaluationDataset

Extracts the metric_input from each sample and constructs an EvaluationDataset. This is used by optimizers to run the metric on the annotated inputs.

from_json() (Lines 644-651)

@classmethod
def from_json(cls, path: str) -> SingleMetricAnnotation

Loads a single metric's annotations from a JSON file. Expects the JSON to have "name" and "samples" keys.

filter() (Lines 653-660)

def filter(self, function: Optional[Callable] = None) -> SingleMetricAnnotation

Returns a new SingleMetricAnnotation containing only samples that satisfy the filter function. If no function is provided, returns all samples.

Example:

accepted_only = annotations.filter(lambda x: x["is_accepted"])

select() (Lines 638-642)

def select(self, indices: List[int]) -> SingleMetricAnnotation

Returns a new SingleMetricAnnotation with only the samples at the specified indices.

sample() (Lines 681-734)

def sample(self, n: int, stratify_key: Optional[str] = None) -> SingleMetricAnnotation

Creates a random subset of n samples. If stratify_key is provided, performs stratified sampling to maintain class proportions.

Raises: ValueError if n exceeds the number of available samples.

stratified_batches() (Lines 761-826)

def stratified_batches(
    self,
    batch_size: int,
    stratify_key: str,
    drop_last_batch: bool = False,
    replace: bool = False,
) -> List[List[SampleAnnotation]]

Creates stratified batches ensuring proportional representation of classes. Used by the GeneticOptimizer during population initialization.

get_prompt_annotations() (Lines 828-837)

def get_prompt_annotations(self) -> Dict[str, List[PromptAnnotation]]

Collects all prompt-level annotations from accepted samples, grouped by prompt name.

Container Protocol

SingleMetricAnnotation supports standard Python container operations:

len(annotations) -- Returns the number of samples.
annotations[i] -- Returns the i-th SampleAnnotation.
for sample in annotations: -- Iterates over SampleAnnotation objects.

Usage Example

from ragas.dataset_schema import MetricAnnotation, SingleMetricAnnotation

# Load from a multi-metric JSON file
all_annotations = MetricAnnotation.from_json("annotations.json", metric_name=None)
single = all_annotations["faithfulness"]

# Or load a single metric directly
single = SingleMetricAnnotation.from_json("faithfulness_annotations.json")

# Inspect
print(f"Metric: {single.name}, Samples: {len(single)}")

# Filter to accepted samples
accepted = single.filter(lambda x: x["is_accepted"])

# Convert to evaluation dataset for running the metric
eval_dataset = single.to_evaluation_dataset()

# Use with optimizer
from ragas.optimizers import GeneticOptimizer
from ragas.losses import BinaryMetricLoss

optimizer = GeneticOptimizer(metric=my_metric, llm=my_llm)
best_prompts = optimizer.optimize(
    dataset=single,
    loss=BinaryMetricLoss(metric="accuracy"),
    config={"population_size": 3},
)

Implements

Principle: Human Annotation Collection

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment