Principle:FlagOpen FlagEmbedding IR Metrics Computation
Overview
A standard set of information retrieval metrics (nDCG, Recall, MRR, MAP, Precision) for evaluating the quality of retrieval and reranking systems against ground truth relevance judgments.
Description
After retrieval/reranking, quality is measured by comparing predicted rankings to ground truth (qrels). Key metrics:
- nDCG@k (normalized discounted cumulative gain, measures ranking quality)
- Recall@k (fraction of relevant documents retrieved)
- MRR@k (mean reciprocal rank of first relevant result)
- MAP@k (mean average precision)
- Precision@k
FlagEmbedding uses pytrec_eval for computation and outputs results as markdown tables or JSON.
Usage
After retrieval/reranking to quantify model performance on benchmarks.
Theoretical Basis
- nDCG = DCG / IDCG, where DCG = Σ(2^rel_i − 1) / log2(i+1).
- Recall@k = |relevant ∩ retrieved@k| / |relevant|.
- MRR = 1/|Q| Σ 1/rank_i.
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment