Implementation:Vibrantlabsai Ragas ContextEntityRecall

Knowledge Sources	Vibrantlabsai_Ragas
Domains	Evaluation, Metrics
Last Updated	2026-02-12 00:00 GMT

Overview

ContextEntityRecall measures the proportion of entities in the ground truth reference that are also found in the retrieved contexts, providing an entity-level recall score for retrieval evaluation.

Description

This metric evaluates context recall at the entity level rather than at the sentence or passage level. It uses an LLM to extract unique entities from both the ground truth reference and the retrieved contexts, then calculates the overlap between these two sets.

The algorithm works in three steps:

1. Entity Extraction: An LLM-based prompt (ExtractEntitiesPrompt) extracts unique named entities from the ground truth text and the concatenated retrieved contexts. The prompt includes few-shot examples covering landmarks, historical events, and dates to guide extraction quality.
2. Set Intersection: The entities from both sources are compared using set intersection.
3. Score Computation: The score is computed as the ratio of the number of entities found in both sets (|CN intersect GN|) to the total number of entities in the ground truth (|GN|). A small epsilon value (1e-8) is added to the denominator to avoid division by zero.

A score of 1.0 indicates that every entity in the ground truth was found in the retrieved contexts. A score closer to 0.0 indicates that the retrieval mechanism failed to capture the relevant entities. This metric is particularly useful in entity-centric use cases such as tourism chatbots, factual QA systems, or knowledge graph-based applications.

Usage

Use this metric when you need to evaluate whether the retrieval step of a RAG pipeline captures the specific named entities mentioned in the ground truth. It is most effective when the quality of retrieval can be measured by entity coverage rather than semantic similarity.

Code Reference

Source Location

• Repository: Vibrantlabsai_Ragas
• File: src/ragas/metrics/_context_entities_recall.py

Signature

@dataclass
class ContextEntityRecall(MetricWithLLM, SingleTurnMetric):
    name: str = "context_entity_recall"
    _required_columns: t.Dict[MetricType, t.Set[str]] = field(
        default_factory=lambda: {
            MetricType.SINGLE_TURN: {"reference", "retrieved_contexts"}
        }
    )
    output_type = MetricOutputType.CONTINUOUS
    context_entity_recall_prompt: PydanticPrompt = field(
        default_factory=ExtractEntitiesPrompt
    )
    max_retries: int = 1

Import

from ragas.metrics import ContextEntityRecall

I/O Contract

Inputs

Name	Type	Required	Description
reference	str	Yes	The ground truth text from which entities are extracted
retrieved_contexts	List[str]	Yes	The list of retrieved context strings from which entities are extracted

Outputs

Name	Type	Description
score	float	A continuous score between 0 and 1 representing the fraction of ground truth entities found in the retrieved contexts

Key Components

ExtractEntitiesPrompt

The ExtractEntitiesPrompt is a Pydantic-based prompt that instructs the LLM to extract unique entities from a given text. It includes four few-shot examples covering diverse entity types such as landmarks (Eiffel Tower, Colosseum, Great Wall of China), dates, people, and historical events (Apollo 11). The prompt explicitly instructs the model to consider different forms or mentions of the same entity as a single entity.

Helper Models

Class	Description
EntitiesList	Pydantic model containing a list of extracted entity strings
ExtractEntitiesPrompt	PydanticPrompt that takes StringIO input and produces EntitiesList output

Usage Examples

Basic Usage

from ragas.metrics import ContextEntityRecall
from ragas.dataset_schema import SingleTurnSample

metric = ContextEntityRecall()
# metric.llm = your_llm_instance

sample = SingleTurnSample(
    reference="Albert Einstein won the Nobel Prize in 1921 for his work on the photoelectric effect.",
    retrieved_contexts=[
        "Einstein, a German-born physicist, received the 1921 Nobel Prize in Physics.",
        "The photoelectric effect was explained by Einstein in 1905."
    ]
)

# score = await metric.single_turn_ascore(sample)

Using the Pre-instantiated Default

from ragas.metrics._context_entities_recall import context_entity_recall

# The module provides a pre-instantiated default:
# context_entity_recall = ContextEntityRecall()

Related Pages

• Environment:Vibrantlabsai_Ragas_Python_3_9_Core_Environment