Implementation:Explodinggradients Ragas SimpleCriteriaScore Metric

Field	Value
source	Repo
domains	Metrics, Evaluation
last_updated	2026-02-10

Overview

SimpleCriteriaScore evaluates submissions against a user-defined criteria definition, returning a discrete score with self-consistency via majority voting.

Description

The SimpleCriteriaScore class judges submissions based on a user-defined criteria string, producing a discrete score. Unlike AspectCritic which returns binary results, SimpleCriteriaScore returns discrete numeric scores. It supports both single-turn and multi-turn evaluation. A self-consistency mechanism via the strictness parameter runs multiple checks and uses majority voting to determine the final score. It inherits from MetricWithLLM, SingleTurnMetric, and MultiTurnMetric.

Key attributes:

• definition -- The criteria string used to evaluate submissions (set at construction and dynamically updatable via property).
• strictness -- Number of self-consistency checks (forced to odd to avoid ties, default 1).
• single_turn_prompt / multi_turn_prompt -- Customizable prompts with the criteria definition injected.

Usage

All input columns are optional. The metric requires an LLM and a definition string to be provided at construction time.

Code Reference

Property	Value
Source Location	src/ragas/metrics/_simple_criteria.py L74-211
Class Signature	class SimpleCriteriaScore(MetricWithLLM, SingleTurnMetric, MultiTurnMetric)
Import	from ragas.metrics import SimpleCriteriaScore

I/O Contract

Inputs

Parameter	Type	Required	Description
user_input	str	Optional	The user query or input
response	str	Optional	The generated response
retrieved_contexts	List[str]	Optional	Retrieved context passages
reference	str	Optional	The reference answer
reference_contexts	List[str]	Optional	Reference context passages

Outputs

Output	Type	Description
score	float	Discrete score determined by the LLM based on the criteria definition

Usage Examples

from ragas.metrics import SimpleCriteriaScore
from ragas.dataset_schema import SingleTurnSample

metric = SimpleCriteriaScore(
    name="helpfulness",
    definition="How helpful is the response in addressing the user's question? Rate from 1 to 5.",
    strictness=3
)
# metric.llm = ...  # Set your LLM

sample = SingleTurnSample(
    user_input="How do I reset my password?",
    response="Go to the settings page and click 'Reset Password'."
)
# score = await metric.single_turn_ascore(sample)

Related Pages

• Explodinggradients_Ragas_AspectCritic_Metric -- Similar criteria-based evaluation but with binary (0/1) output
• Explodinggradients_Ragas_RubricsScore_Metric -- Rubric-based evaluation with structured scoring criteria
• Explodinggradients_Ragas_InstanceRubrics_Metric -- Per-instance rubric evaluation