Implementation:FlagOpen FlagEmbedding LLM Embedder Eval Retrieval

Knowledge Sources	FlagOpen_FlagEmbedding
Domains	Information_Retrieval, Embedding_Models, Evaluation
Last Updated	2026-02-09 00:00 GMT

Overview

General-purpose retrieval evaluation framework supporting dense encoders and BM25 with standard IR metrics.

Description

This is the core retrieval evaluation module that supports various retrieval methods and computes comprehensive metrics:

Supported retrievers:

Dense retrieval: Uses bi-encoder models (query_encoder/key_encoder) with FAISS indexing
BM25: Traditional keyword-based retrieval via Anserini
No retrieval: Baseline without retrieval augmentation

Evaluation workflow: 1. Prepares corpus and queries with optional task-specific instructions 2. Indexes the corpus (builds FAISS index or BM25 index) 3. Performs retrieval to get top-k passages per query 4. Computes metrics: nDCG, Recall, collates retrieved keys, mines negatives

The framework handles result caching (save/load), supports different cutoffs for metrics, and can filter answers from retrieved passages. It's designed to work with the broader LLM-Embedder ecosystem and supports various tasks (QA, chat, ICL, etc.) through the TASK_CONFIG system.

Usage

Use this as the primary retrieval evaluation tool for embedding models on any retrieval task, or as a subroutine called by task-specific evaluation scripts.

Code Reference

Source Location

Repository: FlagOpen_FlagEmbedding
File: research/llm_embedder/evaluation/eval_retrieval.py
Lines: 1-163

Signature

def main(args, accelerator=None, log=True)

Import

from research.llm_embedder.evaluation.eval_retrieval import main as retrieval_main

I/O Contract

Inputs

Name	Type	Required	Description
eval_data	str	Yes	Path to query JSON file
corpus	str	Yes	Path to corpus JSON file
retrieval_method	str	Yes	Method: "dense", "bm25", or "no"
query_encoder	str	No	Dense encoder for queries (if dense method)
key_encoder	str	No	Dense encoder for passages (if dense method)
hits	int	No	Number of passages to retrieve (default: 100)
metrics	List[str]	No	Metrics to compute: ndcg, recall, collate_key, mine_negatives
cutoffs	List[int]	No	Cutoffs for metrics (default: [3, 10, 100])

Outputs

Name	Type	Description
query_ids	List	List of query IDs
preds	List[List[int]]	Predicted passage indices for each query
metrics	Dict	Dictionary with nDCG, Recall, and other metrics
result_file	JSONL	Saved retrieval results (if save_result=True)

Usage Examples

# As standalone script
python research/llm_embedder/evaluation/eval_retrieval.py \
    --eval_data path/to/queries.json \
    --corpus path/to/corpus.json \
    --retrieval_method dense \
    --query_encoder BAAI/llm-embedder \
    --key_encoder BAAI/llm-embedder \
    --hits 100 \
    --metrics ndcg recall collate_key \
    --cutoffs 1 5 10 20 100 \
    --output_dir data/outputs

# As imported function
from research.llm_embedder.evaluation.eval_retrieval import main as retrieval_main
from dataclasses import dataclass

@dataclass
class Args:
    eval_data = "queries.json"
    corpus = "corpus.json"
    retrieval_method = "dense"
    query_encoder = "BAAI/llm-embedder"
    # ... other args

query_ids, preds, metrics = retrieval_main(args=Args())
# metrics: {"ndcg@10": 0.512, "recall@10": 0.734, ...}

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment