Implementation:Run llama Llama index LabelledRagDataset

Overview

This module defines the data structures for labelled RAG (Retrieval-Augmented Generation) evaluation datasets. It includes the example class LabelledRagDataExample for storing individual query-answer pairs with context, the prediction class RagExamplePrediction for storing model outputs, and the dataset classes LabelledRagDataset and RagPredictionDataset for managing collections of examples and predictions.

Source file: llama-index-core/llama_index/core/llama_dataset/rag.py (192 lines)

Class Hierarchy

BaseLlamaExamplePrediction
  └── RagExamplePrediction

BaseLlamaDataExample
  └── LabelledRagDataExample

BaseLlamaPredictionDataset
  └── RagPredictionDataset

BaseLlamaDataset[BaseQueryEngine]
  └── LabelledRagDataset

RagExamplePrediction

class RagExamplePrediction(BaseLlamaExamplePrediction):

Stores the output of a RAG prediction on a single example.

Field	Type	Default	Description
response	str	""	The generated response that can be compared to a reference answer
contexts	Optional[List[str]]	None	The retrieved context texts used to generate the response

The class_name property returns "RagExamplePrediction".

LabelledRagDataExample

class LabelledRagDataExample(BaseLlamaDataExample):

Represents a single labelled RAG evaluation example containing both the "features" (query + context) and the "label" (reference answer).

Field	Type	Default	Description
query	str	""	The user query for the example
query_by	Optional[CreatedBy]	None	Indicates whether the query was generated by a human or AI (with model name)
reference_contexts	Optional[List[str]]	None	The contexts used to generate the reference answer
reference_answer	str	""	The ground-truth answer that would receive full marks upon evaluation
reference_answer_by	Optional[CreatedBy]	None	Indicates whether the reference answer was generated by a human or AI

The class_name property returns "LabelledRagDataExample".

RagPredictionDataset

class RagPredictionDataset(BaseLlamaPredictionDataset):

A dataset for storing collections of RagExamplePrediction instances.

to_pandas

def to_pandas(self) -> Any:

Converts the prediction dataset to a pandas DataFrame with columns:

• response -- the generated responses
• contexts -- the retrieved contexts

Raises ImportError if pandas is not installed. Validates that all predictions are of type RagExamplePrediction.

LabelledRagDataset

class LabelledRagDataset(BaseLlamaDataset[BaseQueryEngine]):

The primary dataset class for RAG evaluation, parameterized with BaseQueryEngine as the predictor type.

to_pandas

def to_pandas(self) -> Any:

Converts the dataset to a pandas DataFrame with columns:

• query
• reference_contexts
• reference_answer
• reference_answer_by
• query_by

Validates that all examples are of type LabelledRagDataExample.

_apredict_example

async def _apredict_example(
    self,
    predictor: BaseQueryEngine,
    example: LabelledRagDataExample,
    sleep_time_in_seconds: int,
) -> RagExamplePrediction:

Asynchronously predicts a single example using a query engine:

1. Sleeps for the specified duration (for rate limiting).
2. Queries the predictor with the example's query.
3. Returns a RagExamplePrediction with the response string and source node texts.

_predict_example

def _predict_example(
    self,
    predictor: BaseQueryEngine,
    example: LabelledRagDataExample,
    sleep_time_in_seconds: int = 0,
) -> RagExamplePrediction:

Synchronous counterpart of _apredict_example. Uses time.sleep for rate limiting and predictor.query for synchronous querying.

_construct_prediction_dataset

def _construct_prediction_dataset(
    self, predictions: Sequence[RagExamplePrediction]
) -> RagPredictionDataset:

Factory method that wraps a sequence of predictions in a RagPredictionDataset.

Aliases

The module provides American English spelling aliases at the bottom:

LabeledRagDataExample = LabelledRagDataExample
LabeledRagDataset = LabelledRagDataset

Dependencies

• llama_index.core.base.base_query_engine.BaseQueryEngine -- used as the predictor type
• llama_index.core.bridge.pydantic.Field -- Pydantic field descriptors
• llama_index.core.llama_dataset.base -- provides all base classes (BaseLlamaDataExample, BaseLlamaDataset, BaseLlamaExamplePrediction, BaseLlamaPredictionDataset, CreatedBy)