Implementation:Vibrantlabsai Ragas ContextPrecision

Knowledge Sources	Vibrantlabsai_Ragas
Domains	LLM Evaluation, RAG Metrics, Context Relevance
Last Updated	2026-02-12 00:00 GMT

Overview

The context precision module provides a family of metrics that evaluate whether the relevant retrieved contexts are ranked higher in the retrieval results, using LLM-based verification, non-LLM string similarity, or direct ID-based comparison.

Description

This module implements multiple variants of context precision, a retrieval quality metric from the RAG (Retrieval-Augmented Generation) evaluation domain. The core idea is to measure Average Precision -- whether contexts that are actually useful for answering a question appear earlier in the retrieval ranking.

The module defines five metric classes:

LLMContextPrecisionWithReference is the base LLM-powered metric. For each retrieved context, it sends the question, context, and reference answer to an LLM via the ContextPrecisionPrompt and receives a binary verdict (1 = useful, 0 = not useful) with a reason. The verdicts are then used to compute the Average Precision score, where higher-ranked relevant contexts contribute more to the final score. It requires columns: user_input, retrieved_contexts, and reference.

LLMContextPrecisionWithoutReference extends the above but uses the model's own response instead of a reference answer as the ground truth for verification. This is useful when no gold-standard reference is available.

NonLLMContextPrecisionWithReference is a non-LLM variant that computes precision by comparing retrieved_contexts against reference_contexts using a configurable string similarity distance measure (defaulting to NonLLMStringSimilarity). Contexts with similarity above a configurable threshold (default 0.5) are considered relevant.

IDBasedContextPrecision is the simplest variant, computing precision by directly comparing retrieved_context_ids against reference_context_ids as a set overlap ratio.

ContextPrecision and ContextUtilization are convenience aliases for the with-reference and without-reference LLM variants, respectively.

The ContextPrecisionPrompt is a PydanticPrompt subclass with three few-shot examples demonstrating useful and non-useful context scenarios. It takes a QAC (question, answer, context) input and returns a Verification (reason, binary verdict) output.

Usage

Import the context precision metrics when evaluating the ranking quality of a retrieval system. Use ContextPrecision (or LLMContextPrecisionWithReference) when you have reference answers and want LLM-based relevance judgments. Use ContextUtilization when you only have the model's response. Use NonLLMContextPrecisionWithReference for faster, LLM-free evaluation with reference contexts. Use IDBasedContextPrecision when your contexts have unique identifiers and you want a simple set-overlap precision score.

Code Reference

Source Location

Repository: Vibrantlabsai_Ragas
File: src/ragas/metrics/_context_precision.py

Signature

@dataclass
class LLMContextPrecisionWithReference(MetricWithLLM, SingleTurnMetric):
    name: str = "llm_context_precision_with_reference"
    context_precision_prompt: PydanticPrompt = field(
        default_factory=ContextPrecisionPrompt
    )
    max_retries: int = 1

    async def _single_turn_ascore(
        self, sample: SingleTurnSample, callbacks: Callbacks
    ) -> float: ...

    async def _ascore(self, row: t.Dict, callbacks: Callbacks) -> float: ...


@dataclass
class LLMContextPrecisionWithoutReference(LLMContextPrecisionWithReference):
    name: str = "llm_context_precision_without_reference"


@dataclass
class NonLLMContextPrecisionWithReference(SingleTurnMetric):
    name: str = "non_llm_context_precision_with_reference"
    distance_measure: SingleTurnMetric = field(
        default_factory=lambda: NonLLMStringSimilarity()
    )
    threshold: float = 0.5

    async def _single_turn_ascore(
        self, sample: SingleTurnSample, callbacks: Callbacks
    ) -> float: ...


@dataclass
class IDBasedContextPrecision(SingleTurnMetric):
    name: str = "id_based_context_precision"

    async def _single_turn_ascore(
        self, sample: SingleTurnSample, callbacks: Callbacks
    ) -> float: ...


@dataclass
class ContextPrecision(LLMContextPrecisionWithReference):
    name: str = "context_precision"


@dataclass
class ContextUtilization(LLMContextPrecisionWithoutReference):
    name: str = "context_utilization"

Import

from ragas.metrics import ContextPrecision, ContextUtilization
from ragas.metrics._context_precision import (
    LLMContextPrecisionWithReference,
    LLMContextPrecisionWithoutReference,
    NonLLMContextPrecisionWithReference,
    IDBasedContextPrecision,
)

I/O Contract

Inputs (LLMContextPrecisionWithReference)

Name	Type	Required	Description
user_input	str	Yes	The user's question or query
retrieved_contexts	List[str]	Yes	Ordered list of retrieved context strings
reference	str	Yes	The ground truth reference answer

Inputs (LLMContextPrecisionWithoutReference)

Name	Type	Required	Description
user_input	str	Yes	The user's question or query
response	str	Yes	The model's generated response (used as reference)
retrieved_contexts	List[str]	Yes	Ordered list of retrieved context strings

Inputs (NonLLMContextPrecisionWithReference)

Name	Type	Required	Description
retrieved_contexts	List[str]	Yes	Ordered list of retrieved context strings
reference_contexts	List[str]	Yes	Ground truth reference context strings

Inputs (IDBasedContextPrecision)

Name	Type	Required	Description
retrieved_context_ids	List[Union[str, int]]	Yes	IDs of retrieved contexts
reference_context_ids	List[Union[str, int]]	Yes	IDs of ground truth relevant contexts

Outputs

Name	Type	Description
score	float	A precision score between 0.0 and 1.0; for LLM variants this is Average Precision reflecting both relevance and ranking quality; for ID-based this is simple set-overlap precision

Usage Examples

Basic Usage with ContextPrecision

from ragas.metrics import ContextPrecision
from ragas.dataset_schema import SingleTurnSample
from ragas.llms import llm_factory

llm = llm_factory("gpt-4o-mini")
metric = ContextPrecision()
metric.llm = llm

sample = SingleTurnSample(
    user_input="What is the capital of France?",
    retrieved_contexts=[
        "Paris is the capital and largest city of France.",
        "France is a country in Western Europe.",
        "The Eiffel Tower is located in Paris.",
    ],
    reference="Paris is the capital of France.",
)

score = await metric.single_turn_ascore(sample)
print(f"Context Precision: {score}")

Using ContextUtilization (No Reference)

from ragas.metrics import ContextUtilization
from ragas.dataset_schema import SingleTurnSample

metric = ContextUtilization()
metric.llm = llm

sample = SingleTurnSample(
    user_input="What is the capital of France?",
    retrieved_contexts=[
        "Paris is the capital and largest city of France.",
        "France is a country in Western Europe.",
    ],
    response="The capital of France is Paris.",
)

score = await metric.single_turn_ascore(sample)
print(f"Context Utilization: {score}")

Using ID-Based Precision

from ragas.metrics._context_precision import IDBasedContextPrecision
from ragas.dataset_schema import SingleTurnSample

metric = IDBasedContextPrecision()

sample = SingleTurnSample(
    retrieved_context_ids=["doc1", "doc2", "doc3"],
    reference_context_ids=["doc1", "doc3", "doc5"],
)

score = await metric.single_turn_ascore(sample)
print(f"ID-Based Precision: {score}")  # 2/3 = 0.667

Average Precision Calculation

The LLM-based variants compute Average Precision using the following algorithm:

cumsum = 0
numerator = 0.0
for i, verdict in enumerate(verifications):
    v = 1 if verdict else 0
    cumsum += v
    if v:
        numerator += cumsum / (i + 1)
score = numerator / (cumsum + 1e-10)

This rewards relevant contexts that appear earlier in the ranking. A score of 1.0 means all relevant contexts are ranked at the top; a score close to 0.0 means relevant contexts are buried deep in the results.

Module-Level Instances

The module pre-instantiates two convenience objects:

context_precision = ContextPrecision()
context_utilization = ContextUtilization()

These can be imported directly and used after setting the llm attribute.

Related Pages

PydanticPrompt - Base prompt class used by ContextPrecisionPrompt
MetricWithLLM - Mixin providing LLM integration for metrics
SingleTurnMetric - Base class for single-turn evaluation metrics
NonLLMStringSimilarity - Default distance measure for the non-LLM variant
Vibrantlabsai_Ragas_FactualCorrectness - Another Ragas evaluation metric

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment