Implementation:Arize ai Phoenix FaithfulnessEvaluator

Overview

FaithfulnessEvaluator is an LLM-based classification evaluator in the arize-phoenix-evals package that detects whether a model's response is faithful to a provided context. It extends ClassificationEvaluator and uses a judge LLM to classify responses as faithful or unfaithful relative to grounding context.

Description

The FaithfulnessEvaluator assesses whether a language model's output is supported by and consistent with the provided reference context. This is a critical metric for retrieval-augmented generation (RAG) systems, where responses should be grounded in retrieved documents rather than fabricated.

The evaluator sends the input query, model output, and reference context through a prompt template loaded from a pre-generated configuration (FAITHFULNESS_CLASSIFICATION_EVALUATOR_CONFIG). The LLM judge then classifies the response and provides an explanation.

This evaluator is the recommended replacement for the deprecated HallucinationEvaluator, using updated terminology:

• "faithful" / "unfaithful" labels instead of "factual" / "hallucinated".
• Maximizes score (1.0 = faithful) instead of minimizing it.

Parameter	Type	Description
llm	LLM	The LLM instance to use as the judge for evaluation. Must support tool calling or structured output.

Usage

from phoenix.evals.metrics import FaithfulnessEvaluator
from phoenix.evals import LLM

llm = LLM(provider="openai", model="gpt-4o-mini")
evaluator = FaithfulnessEvaluator(llm=llm)

Code Reference

Property	Value
Source File	packages/phoenix-evals/src/phoenix/evals/metrics/faithfulness.py
Module	phoenix.evals.metrics.faithfulness
Class	FaithfulnessEvaluator(ClassificationEvaluator)
Lines	~68
Kind	"llm"
Direction	"maximize"
Domain	LLM Evaluation, Metrics

Class Attributes

Attribute	Description
NAME	The evaluator name, loaded from FAITHFULNESS_CLASSIFICATION_EVALUATOR_CONFIG.name.
PROMPT	A PromptTemplate built from the config's messages.
CHOICES	Classification labels (faithful, unfaithful) from the config.
DIRECTION	Optimization direction from the config.

Input Schema

Defined by the inner class FaithfulnessInputSchema(BaseModel):

Field	Type	Description
input	str	The input query.
output	str	The response to the query.
context	str	The context or reference text against which faithfulness is evaluated.

I/O Contract

Input

Field	Type	Required	Description
input	str	Yes	The original query or question.
output	str	Yes	The model's response to be evaluated for faithfulness.
context	str	Yes	The reference context or document(s) that the response should be grounded in.

Output

Returns a list containing one Score object with the following fields:

Field	Description
name	"faithfulness"
score	1.0 if faithful, 0.0 if unfaithful.
label	The classification label ("faithful" or "unfaithful").
explanation	An explanation from the LLM judge.
metadata	Dictionary containing the model name used for evaluation.
kind	"llm"
direction	"maximize"

Usage Examples

Evaluating a Faithful Response

from phoenix.evals.metrics.faithfulness import FaithfulnessEvaluator
from phoenix.evals import LLM

llm = LLM(provider="openai", model="gpt-4o-mini")
faithfulness_eval = FaithfulnessEvaluator(llm=llm)

eval_input = {
    "input": "What is the capital of France?",
    "output": "Paris is the capital of France.",
    "context": "Paris is the capital and largest city of France.",
}
scores = faithfulness_eval.evaluate(eval_input)
print(scores)
# [Score(name='faithfulness', score=1.0, label='faithful',
#     explanation='Information is supported by context',
#     metadata={'model': 'gpt-4o-mini'},
#     kind="llm", direction="maximize")]

Detecting an Unfaithful Response

eval_input = {
    "input": "What is the population of Paris?",
    "output": "Paris has a population of 10 million people.",
    "context": "Paris has a population of approximately 2.1 million in the city proper.",
}
scores = faithfulness_eval.evaluate(eval_input)
# Expected: score=0.0, label='unfaithful'

Related Pages

• Implementation:Arize_ai_Phoenix_HallucinationEvaluator -- Deprecated predecessor using "factual"/"hallucinated" labels.
• Implementation:Arize_ai_Phoenix_CorrectnessEvaluator -- LLM-based correctness evaluation (without context grounding).
• Implementation:Arize_ai_Phoenix_DocumentRelevanceEvaluator -- Evaluates whether a document is relevant to a query.
• Implementation:Arize_ai_Phoenix_Evals_Public_API -- The top-level phoenix.evals public API surface.