Implementation:Explodinggradients Ragas RankingMetric Class

Field	Value
source	Explodinggradients_Ragas (GitHub)
domains	Metrics, Ranking
last_updated	2026-02-10 00:00 GMT

Overview

The RankingMetric class and ranking_metric decorator provide a framework for evaluating metrics that produce ranked lists of items, such as search result ordering or relevance prioritization.

Description

RankingMetric is a dataclass that extends SimpleLLMMetric and RankingValidator. It automatically creates a structured response model with reason and value fields at initialization using the instructor library for structured LLM outputs. The class supports correlation calculation between gold labels and predictions using Cohen's kappa score with quadratic weighting from scikit-learn. The allowed_values attribute specifies the expected number of items in the ranking list. The companion ranking_metric decorator function transforms a regular function into a RankingMetric instance, using create_metric_decorator internally.

Usage

Create a RankingMetric directly by providing a name, LLM, prompt, and the expected number of ranked items. Alternatively, use the @ranking_metric decorator to transform a function into a ranking metric. Use RankingMetric.load() to restore a previously saved metric from a JSON file.

Code Reference

Item	Detail
Source Location	src/ragas/metrics/ranking.py L18-187
Class Signature	class RankingMetric(SimpleLLMMetric, RankingValidator)
Decorator Signature	def ranking_metric(*, name: Optional[str] = None, allowed_values: Optional[int] = None, **metric_params) -> Callable
Import	from ragas.metrics.ranking import RankingMetric, ranking_metric

I/O Contract

Inputs

Parameter	Type	Description
name	str	Name of the ranking metric
llm	Optional[BaseRagasLLM]	Language model instance for evaluation
prompt	Optional[Union[str, Prompt]]	Prompt template with placeholders for evaluation inputs
allowed_values	int	Expected number of items in the ranking list (default 2)

Outputs

Output	Type	Description
result.value	List[str]	Ranked list of items produced by the LLM
result.reason	str	Reasoning for the ranking order
get_correlation()	float	Cohen's kappa score between gold labels and predictions

Usage Examples

from ragas.metrics.ranking import RankingMetric
from ragas.llms import llm_factory
from openai import OpenAI

client = OpenAI(api_key="your-api-key")
llm = llm_factory("gpt-4o-mini", client=client)

metric = RankingMetric(
    name="relevance_ranking",
    llm=llm,
    prompt="Rank these results by relevance: {results}",
    allowed_values=3,
)

result = metric.score(llm=llm, results="result1, result2, result3")
print(result.value)  # A list of 3 ranked items

from ragas.metrics.ranking import ranking_metric

@ranking_metric(name="priority_ranker", allowed_values=3)
def rank_by_urgency(user_input: str, responses: list) -> list:
    """Rank responses by urgency keywords."""
    urgency_keywords = ["urgent", "asap", "critical"]
    scored = [(sum(kw in r.lower() for kw in urgency_keywords), r) for r in responses]
    ranked = sorted(scored, key=lambda x: x[0], reverse=True)
    return [item[1] for item in ranked[:3]]

Related Pages

• MetricResult_Class - Result wrapper returned by metric scoring
• MetricValidators_Module - RankingValidator used by this class
• SimplePrompt_Class - Prompt class used for metric prompts