Implementation:FlagOpen FlagEmbedding Compute Metrics QA Recall
| Knowledge Sources | |
|---|---|
| Domains | Information Retrieval, Question Answering, Evaluation Metrics |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A utility module for evaluating QA recall metrics by checking if retrieved documents contain answer strings.
Description
This module provides functions to evaluate the recall performance of retrieval systems in Question Answering tasks. It implements a token-based answer matching approach, checking whether retrieved documents contain the correct answers. The core functionality includes a simple tokenizer based on Unicode character classes, answer checking logic that performs case-insensitive token matching, and the Recall@k metric computation for evaluating retrieval quality at different cutoff positions. The implementation is referenced from Facebook's Contriever project and uses Unicode normalization for robust text matching.
Usage
Use this module when evaluating the performance of document retrieval systems for Question Answering tasks. It is particularly useful for measuring how many questions have at least one correct answer in the top-k retrieved documents. The module is designed for tutorial and evaluation purposes in the FlagEmbedding framework, specifically for the evaluation tutorial section.
Code Reference
Source Location
- Repository: FlagOpen_FlagEmbedding
- File: Tutorials/4_Evaluation/utils/compute_metrics.py
- Lines: 1-95
Signature
<syntaxhighlight lang="python"> def evaluate_qa_recall(ctxs, answers, k_values: Union[int, List[int]]=100):
"""Compute Recall@k for QA task"""
class SimpleTokenizer:
"""A simple regex-based tokenizer for text"""
def __init__(self):
pass
def tokenize(self, text, uncased=False):
pass
def has_answer(answers, text, tokenizer) -> bool:
"""Check if a document contains an answer string"""
def check_answer(example, tokenizer) -> List[bool]:
"""Search through all the top docs to see if they have any of the answers"""