Overview
DocumentRelevanceEvaluator is an LLM-based classification evaluator in the arize-phoenix-evals package that determines whether a given document is relevant to answering a specific question. It extends ClassificationEvaluator and uses a judge LLM to classify document-query pairs as relevant or unrelated.
Description
The DocumentRelevanceEvaluator is designed for retrieval-augmented generation (RAG) evaluation pipelines. It assesses whether a retrieved document contains information that is useful for answering a user's query. This is a key metric for evaluating the quality of retrieval systems.
The evaluator loads its configuration from DOCUMENT_RELEVANCE_CLASSIFICATION_EVALUATOR_CONFIG, which provides the prompt template, classification choices, and optimization direction. It sends the input query and document text to an LLM judge for classification.
| Parameter |
Type |
Description
|
llm |
LLM |
The LLM instance to use as the judge for evaluation. Must support tool calling or structured output.
|
Usage
from phoenix.evals.metrics import DocumentRelevanceEvaluator
from phoenix.evals import LLM
llm = LLM(provider="openai", model="gpt-4o-mini")
evaluator = DocumentRelevanceEvaluator(llm=llm)
Code Reference
Class Attributes
| Attribute |
Description
|
NAME |
The evaluator name, loaded from DOCUMENT_RELEVANCE_CLASSIFICATION_EVALUATOR_CONFIG.name.
|
PROMPT |
A PromptTemplate built from the config's messages.
|
CHOICES |
Classification labels (relevant, unrelated) from the config.
|
DIRECTION |
Optimization direction from the config.
|
Input Schema
Defined by the inner class DocumentRelevanceInputSchema(BaseModel):
| Field |
Type |
Description
|
input |
str |
The input query.
|
document_text |
str |
The document being evaluated for relevance.
|
I/O Contract
Input
| Field |
Type |
Required |
Description
|
input |
str |
Yes |
The user query or question.
|
document_text |
str |
Yes |
The text content of the document to evaluate for relevance.
|
Output
Returns a list containing one Score object with the following fields:
| Field |
Description
|
name |
The evaluator name (e.g., "document_relevance").
|
score |
1.0 if relevant, 0.0 if unrelated.
|
label |
The classification label ("relevant" or "unrelated").
|
explanation |
An explanation from the LLM judge.
|
metadata |
Dictionary containing the model name used for evaluation.
|
kind |
"llm"
|
direction |
The optimization direction (maximize).
|
Usage Examples
Evaluating a Relevant Document
from phoenix.evals.metrics.document_relevance import DocumentRelevanceEvaluator
from phoenix.evals import LLM
llm = LLM(provider="openai", model="gpt-4o-mini")
relevance_eval = DocumentRelevanceEvaluator(llm=llm)
eval_input = {
"input": "What is the capital of France?",
"document_text": "Paris is the capital and largest city of France.",
}
scores = relevance_eval.evaluate(eval_input)
print(scores)
# Expected: score=1.0, label='relevant'
Detecting an Unrelated Document
eval_input = {
"input": "What is the capital of France?",
"document_text": "The Amazon rainforest covers much of northwestern Brazil.",
}
scores = relevance_eval.evaluate(eval_input)
# Expected: score=0.0, label='unrelated'
Batch Evaluation with a DataFrame
from phoenix.evals import evaluate_dataframe, LLM
from phoenix.evals.metrics import DocumentRelevanceEvaluator
import pandas as pd
llm = LLM(provider="openai", model="gpt-4o-mini")
evaluator = DocumentRelevanceEvaluator(llm=llm)
df = pd.DataFrame({
"input": ["What is photosynthesis?", "How does gravity work?"],
"document_text": [
"Photosynthesis is the process by which plants convert sunlight into energy.",
"The French Revolution began in 1789.",
],
})
results = evaluate_dataframe(df, evaluator)
Related Pages
