Implementation:Evidentlyai Evidently Legacy LLM Judge Feature
| Knowledge Sources | |
|---|---|
| Domains | ML Monitoring, LLM Evaluation, AI Judges |
| Last Updated | 2026-02-14 12:00 GMT |
Overview
Provides a generated features class that uses a configurable LLM (Large Language Model) as a judge to evaluate text data, supporting binary classification, multiclass classification, and custom prompt templates.
Description
The LLMJudge class extends GeneratedFeatures (note the plural -- it can produce multiple output columns) to evaluate text data using an LLM provider. It wraps the LLM interaction through Evidently's LLMWrapper abstraction, which is obtained via get_llm_wrapper using the specified provider and model.
Key design aspects:
- Provider and model configuration: The class accepts a provider string (e.g., "openai") and a model string (e.g., "gpt-4") to select the LLM backend.
- Input column mapping: Supports either a single input_column or a dictionary of input_columns mapping DataFrame column names to template variable names. If input_column is used, it is mapped to the default variable name "input".
- Prompt templates: The template parameter accepts a BaseLLMPromptTemplate or one of its subclasses (BinaryClassificationPromptTemplate, MulticlassClassificationPromptTemplate). The template controls how input data is formatted into LLM prompts and how responses are parsed.
- Batch processing: The generate_features method uses run_batch_sync on the LLM wrapper to process all rows in a single batch call, iterating messages via the template's iterate_messages method.
- Multiple output columns: The list_columns method returns a list of ColumnName objects based on the template's output columns. The column types are determined by the template's get_type method.
The _llm_wrapper is lazily initialized as a private attribute and cached for reuse.
The module also re-exports BinaryClassificationPromptTemplate, MulticlassClassificationPromptTemplate, BaseLLMPromptTemplate, Uncertainty, LLMMessage, and LLMWrapper for convenience.
Usage
Use this feature when you need an LLM to evaluate or classify text data as part of an Evidently monitoring pipeline. Common use cases include evaluating response quality, detecting harmful content, classifying intents, or performing any custom evaluation that can be expressed as a prompt template. This is a legacy feature class for use with Evidently's legacy report framework.
Code Reference
Source Location
- Repository: Evidentlyai_Evidently
- File: src/evidently/legacy/features/llm_judge.py
Signature
class LLMJudge(GeneratedFeatures):
class Config:
type_alias = "evidently:feature:LLMJudge"
DEFAULT_INPUT_COLUMN: ClassVar = "input"
provider: str
model: str
input_column: Optional[str] = None
input_columns: Optional[Dict[str, str]] = None
template: BaseLLMPromptTemplate
def get_llm_wrapper(self, options: Options) -> LLMWrapper: ...
def get_input_columns(self): ...
def generate_features(self, data: pd.DataFrame, data_definition: DataDefinition, options: Options) -> pd.DataFrame: ...
def list_columns(self) -> List[ColumnName]: ...
def get_type(self, subcolumn: Optional[str] = None) -> ColumnType: ...
Import
from evidently.legacy.features.llm_judge import LLMJudge
from evidently.legacy.features.llm_judge import BinaryClassificationPromptTemplate
from evidently.legacy.features.llm_judge import MulticlassClassificationPromptTemplate
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| provider | str | Yes | LLM provider identifier (e.g., "openai") |
| model | str | Yes | Model name or identifier (e.g., "gpt-4") |
| input_column | Optional[str] | No | Single input column name (mutually exclusive with input_columns) |
| input_columns | Optional[Dict[str, str]] | No | Mapping of DataFrame column names to template variable names |
| template | BaseLLMPromptTemplate | Yes | Prompt template defining how to format inputs and parse outputs |
Outputs
| Name | Type | Description |
|---|---|---|
| return | pd.DataFrame | DataFrame with one or more columns produced by the LLM evaluation, as defined by the template's output columns |
Usage Examples
from evidently.legacy.features.llm_judge import (
LLMJudge,
BinaryClassificationPromptTemplate,
)
# Binary classification judge
judge = LLMJudge(
provider="openai",
model="gpt-4",
input_column="response",
template=BinaryClassificationPromptTemplate(
target_category="relevant",
non_target_category="irrelevant",
pre_messages=[],
criteria="Is the response relevant to the question?",
include_reasoning=True,
),
display_name="Relevance Judge",
)
# Multi-column input
judge_multi = LLMJudge(
provider="openai",
model="gpt-4",
input_columns={"question": "input", "answer": "output"},
template=BinaryClassificationPromptTemplate(
target_category="correct",
non_target_category="incorrect",
pre_messages=[],
criteria="Is the answer correct for the given question?",
include_reasoning=False,
),
)