Implementation:FlagOpen FlagEmbedding AbsEvaluator Compute Metrics

Signature

@staticmethod
def compute_metrics(
    qrels: Dict[str, Dict[str, int]],
    search_results: Dict[str, Dict[str, float]],
    k_values: List[int],
) -> dict:

Import:

from FlagEmbedding.abc.evaluation.evaluator import AbsEvaluator

Internal Behavior

Internally calls evaluate_metrics(), evaluate_mrr(), and evaluate_recall_cap() using pytrec_eval.

I/O

Input:

qrels — {qid: {docid: relevance_int}}
search_results — {qid: {docid: score_float}}
k_values — [1, 3, 5, 10, 100, 1000]

Output: dict with keys like ndcg_at_10, recall_at_10, mrr_at_10, map_at_10, precision_at_10, recall_cap_at_10.

AbsEvalRunner.evaluate_metrics

Also related is AbsEvalRunner.evaluate_metrics() at runner.py:L137-181, which collects results across datasets and outputs to markdown/JSON.

@staticmethod
def evaluate_metrics(
    search_results_save_dir: str,
    output_method: str = "markdown",
    output_path: str = "./eval_dev_results.md",
    metrics: Union[str, List[str]] = ["ndcg_at_10", "recall_at_10"]
):

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment