Principle:Explodinggradients Ragas Human Annotation Collection

Human Annotation Collection

Human Annotation Collection is a principle in the Ragas evaluation toolkit that defines how human-annotated evaluation data is collected, structured, and used to drive metric prompt optimization.

Motivation

Prompt optimization algorithms (see Genetic Prompt Optimization and DSPy Prompt Optimization) are supervised processes: they require a ground-truth signal indicating what the metric should produce for each input. Human annotations provide this signal. Without a structured annotation format, it would be impossible to train, evaluate, or compare different prompt candidates in a principled way.

Theoretical Foundation

Ground Truth from Human Judgment

The fundamental assumption is that human annotators can provide authoritative labels for evaluation metrics. For a given metric input (which may include a user query, LLM response, reference answer, and retrieved contexts), a human annotator determines:

• Whether to accept the metric's output -- Did the metric produce a reasonable result?
• What the correct output should be -- If the metric's output was wrong, what should it have been?
• How the intermediate prompts should have responded -- For metrics with multiple prompt stages, what should each prompt's output have been?

Structured Annotation Format

Annotations are organized in a three-level hierarchy:

1. MetricAnnotation -- A collection of annotations grouped by metric name. This is the top-level container that can hold annotations for multiple metrics in a single file.
2. SingleMetricAnnotation -- All annotations for a single metric, consisting of a metric name and a list of sample annotations. This is the unit consumed by optimizers.
3. SampleAnnotation -- A single annotated sample containing:
   • metric_input -- The input fields passed to the metric (e.g., user_input, response, reference, retrieved_contexts).
   • metric_output -- The metric's original predicted output (a float value).
   • prompts -- A dictionary of prompt-level annotations, each containing the prompt input, the prompt's original output, and optionally an edited output reflecting what the human believes the prompt should have produced.
   • is_accepted -- A boolean indicating whether the human accepted the metric's output for this sample.
   • target -- An optional target value when the human provides a specific numeric score.

Prompt-Level Annotations

A key feature of the annotation format is that it captures not just the final metric output but also the intermediate prompt-level behavior. Each PromptAnnotation records:

• prompt_input -- The exact input dictionary passed to the prompt.
• prompt_output -- The original output dictionary produced by the prompt.
• edited_output -- The human-corrected output, if the original was wrong.

This fine-grained annotation enables the optimizer to target specific prompts within a multi-step metric, rather than treating the entire metric as a black box.

Accepted vs. Rejected Samples

The is_accepted flag creates two categories of annotations:

• Accepted samples (where is_accepted=True) indicate that the metric produced the correct output. These serve as positive examples during optimization.
• Rejected samples (where is_accepted=False) indicate that the metric produced an incorrect output. For binary metrics, the optimizer automatically inverts the metric output to derive the correct label. These serve as negative examples that help the optimizer understand failure modes.

Both categories contribute to the optimization process, providing complementary training signals.

Data Loading

Annotations can be loaded from JSON files or constructed programmatically:

• From JSON -- A JSON file containing a dictionary of metric names to lists of sample annotations.
• Filtering by metric -- When loading, a specific metric name can be provided to extract only the relevant annotations.
• Programmatic construction -- SingleMetricAnnotation can be instantiated directly with a name and list of SampleAnnotation objects.

Dataset Operations

The annotation format supports several operations needed by optimizers:

• Filtering -- Select samples by criteria (e.g., only accepted samples).
• Sampling -- Draw random subsets with optional stratification by metric output.
• Stratified batching -- Create batches with proportional class representation.
• Conversion to EvaluationDataset -- Extract the metric inputs as a standard EvaluationDataset for running the metric.
• Prompt annotation extraction -- Collect all prompt-level annotations grouped by prompt name.

Minimum Data Requirements

The GeneticOptimizer requires a minimum of 10 annotated samples. In general, more annotations lead to better optimization results because:

• More positive examples provide richer signal for reverse-engineering instructions.
• More negative examples expose more failure modes for feedback mutation.
• Larger datasets produce more reliable fitness estimates.

Implemented By

• Implementation: MetricAnnotation Class

See Also

• Implementation:Explodinggradients_Ragas_MetricAnnotation_Class
• Genetic Prompt Optimization -- Consumes annotations for fitness-driven prompt search.
• DSPy Prompt Optimization -- Consumes annotations for MIPROv2 training.
• Optimization Loss Functions -- Uses annotations to compute loss values.
• Metric Baseline Correlation -- Uses annotations to measure baseline agreement.