Implementation:Arize ai Phoenix Legacy Retrievals

Overview

The Legacy Retrievals module provides utility functions for evaluating the retrieval step of retrieval-augmented generation (RAG) pipelines. It implements compute_precisions_at_k() for calculating precision at various ranks from a list of relevance classifications, and classify_relevance() for determining whether a single document is relevant to a query using the OpenAI API.

compute_precisions_at_k() is a pure computation function that takes an ordered list of boolean relevance labels (with possible None values for unknown classifications) and returns precision@k values for each position k. This enables evaluation of how well a retrieval system ranks relevant documents.

classify_relevance() is a standalone LLM-based classifier that directly invokes the OpenAI Chat Completions API to determine document relevance, returning True (relevant), False (irrelevant), or None (unparseable output). Unlike the broader llm_classify() framework, this function is a self-contained utility with its own built-in prompt template and system message.

Code Reference

Attribute	Details
Source File	`packages/phoenix-evals/src/phoenix/evals/legacy/retrievals.py`
Repository	Arize-ai/phoenix
Lines	89
Module	`phoenix.evals.legacy.retrievals`
Key Symbols	`compute_precisions_at_k()`, `classify_relevance()`
Dependencies	`openai` (imported lazily inside `classify_relevance()`)

I/O Contract

compute_precisions_at_k()

Parameter	Type	Description
`relevance_classifications`	`List[Optional[bool]]`	Ordered list of relevance labels for retrieved documents. `True` = relevant, `False` = irrelevant, `None` = unknown (omitted from calculation).
Returns	`List[Optional[float]]`	Precision@k values for k = 1, 2, ..., n. Returns `None` at positions where all classifications so far are `None`.

Algorithm: Iterates through the classification list, maintaining running counts of relevant and non-None classifications. At each position k, precision@k = (number of relevant) / (number of non-None classifications up to k).

classify_relevance()

Parameter	Type	Description
`query`	`str`	The search query text.
`document`	`str`	The retrieved document text.
`model_name`	`str`	The OpenAI model name to use (e.g., `"gpt-4"`).
Returns	`Optional[bool]`	`True` if relevant, `False` if irrelevant, `None` if the output could not be parsed.

The function uses a hardcoded system message instructing the LLM to respond with only "relevant" or "irrelevant", and a prompt template with {query} and {reference} placeholders.

Internal Constants

Constant	Description
`_EVALUATION_SYSTEM_MESSAGE`	System prompt instructing the LLM to classify relevance as "relevant" or "irrelevant".
`_QUERY_CONTEXT_PROMPT_TEMPLATE`	User prompt template with `{query}` and `{reference}` placeholders.

Usage Examples

from phoenix.evals.legacy.retrievals import (
    compute_precisions_at_k,
    classify_relevance,
)

# Compute precision@k from relevance labels
classifications = [True, False, True, None, True]
precisions = compute_precisions_at_k(classifications)
# precisions = [1.0, 0.5, 0.667, 0.667, 0.75]
# Position 1: 1/1 = 1.0
# Position 2: 1/2 = 0.5
# Position 3: 2/3 = 0.667
# Position 4: 2/3 = 0.667 (None is skipped)
# Position 5: 3/4 = 0.75

# Classify document relevance using OpenAI
is_relevant = classify_relevance(
    query="What is machine learning?",
    document="Machine learning is a subset of AI that enables systems to learn from data.",
    model_name="gpt-4",
)
# is_relevant = True

Related Pages

Arize_ai_Phoenix_Legacy_Classify - General-purpose classification framework; classify_relevance() is a specialized standalone alternative
Arize_ai_Phoenix_Legacy_Default_Templates - RAG_RELEVANCY_PROMPT_TEMPLATE provides a more feature-rich relevance evaluation template
Arize_ai_Phoenix_Legacy_Utils - snap_to_rail() utility for constrained output parsing

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment