Implementation:Marker Inc Korea AutoRAG Select Best

Knowledge Sources	AutoRAG
Domains	Evaluation Strategy, RAG Pipeline Optimization
Last Updated	2026-02-12 00:00 GMT

Overview

Concrete tool for selecting the best-performing module from multiple candidates based on evaluation metrics, provided by the AutoRAG framework.

Description

The select_best function is the strategy dispatch entry point that routes to one of three concrete selection algorithms: select_best_average (mean), select_best_rr (reciprocal rank), or select_normalize_mean (normalized mean). Each algorithm receives a list of result DataFrames (one per module candidate), the metric column names to evaluate, and optional metadata associated with each candidate. The function returns a tuple of the best candidate's result DataFrame and its associated metadata.

The mean strategy computes the grand average of all specified metric columns across all rows for each candidate and selects the one with the highest average. The rank strategy ranks candidates independently on each metric's column-wise mean, computes reciprocal ranks, and selects the candidate with the highest sum of reciprocal ranks. The normalize_mean strategy applies min-max normalization to each metric's column-wise mean across all candidates before summing, ensuring equal weighting regardless of metric scale.

Usage

Import select_best when implementing custom node evaluation logic or when you need to programmatically compare module outputs outside of the standard AutoRAG trial workflow. Within the standard workflow, this function is called internally by each node's evaluation step to determine which module candidate wins.

Code Reference

Source Location

Repository: AutoRAG
File: autorag/strategy.py (lines 95-165)

Signature

def select_best(
    results: List[pd.DataFrame],
    columns: Iterable[str],
    metadatas: Optional[List[Any]] = None,
    strategy_name: str = "mean",
) -> Tuple[pd.DataFrame, Any]:

Import

from autorag.strategy import select_best

I/O Contract

Inputs

Name	Type	Required	Description
results	List[pd.DataFrame]	yes	List of result DataFrames, one per module candidate. Each DataFrame must contain the columns specified in the columns parameter.
columns	Iterable[str]	yes	Names of the metric columns to use for comparison (e.g., ["retrieval_f1", "retrieval_recall"]). All specified columns must exist in every result DataFrame.
metadatas	Optional[List[Any]]	no	List of metadata objects associated with each result DataFrame (e.g., module configuration dicts). Must have the same length as results. Defaults to a list of None values.
strategy_name	str	no	Name of the selection strategy to use. Must be one of "mean", "rank", or "normalize_mean". Default is "mean".

Outputs

Name	Type	Description
best_result	pd.DataFrame	The result DataFrame of the best-performing module candidate as determined by the selected strategy.
best_metadata	Any	The metadata object associated with the best-performing candidate. Returns None if no metadatas were provided.

Usage Examples

Basic Usage

import pandas as pd
from autorag.strategy import select_best

# Simulated results from three retrieval module candidates
results = [
    pd.DataFrame({"retrieval_f1": [0.8, 0.7, 0.9], "retrieval_recall": [0.85, 0.75, 0.88]}),
    pd.DataFrame({"retrieval_f1": [0.6, 0.65, 0.7], "retrieval_recall": [0.9, 0.92, 0.91]}),
    pd.DataFrame({"retrieval_f1": [0.75, 0.78, 0.8], "retrieval_recall": [0.82, 0.84, 0.86]}),
]
metadatas = [
    {"module_type": "bm25", "top_k": 10},
    {"module_type": "vectordb", "embedding_model": "openai"},
    {"module_type": "hybrid_rrf", "weight_range": "(4, 80)"},
]

# Select using mean strategy
best_df, best_meta = select_best(results, ["retrieval_f1", "retrieval_recall"], metadatas, strategy_name="mean")
print(f"Best module: {best_meta}")

Using Reciprocal Rank Strategy

from autorag.strategy import select_best

# Use rank-based selection when metrics have different scales
best_df, best_meta = select_best(
    results=results,
    columns=["retrieval_f1", "retrieval_recall"],
    metadatas=metadatas,
    strategy_name="rank",
)
print(f"Best module (by reciprocal rank): {best_meta}")

Using Normalized Mean Strategy

from autorag.strategy import select_best

# Use normalized mean when metrics have very different numerical ranges
best_df, best_meta = select_best(
    results=results,
    columns=["retrieval_f1", "retrieval_recall"],
    metadatas=metadatas,
    strategy_name="normalize_mean",
)
print(f"Best module (by normalized mean): {best_meta}")

Related Pages

Implements Principle

Principle:Marker_Inc_Korea_AutoRAG_Strategy_Selection

Requires Environment

Uses Heuristic

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment