Implementation:FlagOpen FlagEmbedding Reinforced IR Get Prompts

Knowledge Sources	FlagOpen_FlagEmbedding
Domains	Natural Language Processing, Information Retrieval, Data Generation
Last Updated	2026-02-09 00:00 GMT

Overview

A comprehensive prompt generation system for query generation (QG), query quality control (QC), and reranking tasks across multiple information retrieval datasets.

Description

This module provides prompt templates and metadata dictionaries for generating synthetic queries from passages, evaluating query-passage relevance, and reranking documents. It supports 25+ different datasets including MS MARCO, Natural Questions, HotpotQA, SciDocs, FEVER, and domain-specific tasks like ArXiv, finance, healthcare, and law. The system includes configurable query styles (plain/simple to professional/complex), variable query lengths, and task-specific instructions for different retrieval scenarios (query-passage, claim-document, question-answer, etc.).

The module implements three main prompt generation functions:

get_query_generation_prompt: Creates prompts for generating queries from passages with optional few-shot examples
get_additional_info_generation_prompt: Generates brief answers for queries to augment training data
get_quality_control_prompt: Validates query-passage relevance using binary classification

Usage

Use this module when you need to generate synthetic training data for information retrieval models, particularly for reinforcement learning approaches where you need diverse, high-quality query-passage pairs with explicit relevance judgments.

Code Reference

Source Location

Repository: FlagOpen_FlagEmbedding
File: research/Reinforced_IR/data_generation/prompts/get_prompts.py
Lines: 1-755

Signature

def get_query_generation_prompt(dataset_name: str, passage: str, use_examples: bool) -> str:
    """Generate a prompt for query generation based on dataset and passage"""

def get_additional_info_generation_prompt(dataset_name: str, query: str) -> str:
    """Generate a prompt for creating brief answers to queries"""

def get_quality_control_prompt(dataset_name: str, query: str, passage: str) -> str:
    """Generate a prompt for evaluating query-passage relevance"""

def get_reranker_prompt(dataset_name: str, query: str, passage: str) -> str:
    """Generate a prompt for reranking query-passage pairs on 0-4 scale"""

Import

from research.Reinforced_IR.data_generation.prompts.get_prompts import (
    get_query_generation_prompt,
    get_additional_info_generation_prompt,
    get_quality_control_prompt,
    get_reranker_prompt,
    TASK_DICT,
    QUERY_TYPE_DICT,
    PASSAGE_TYPE_DICT
)

I/O Contract

Inputs

Name	Type	Required	Description
dataset_name	str	Yes	Dataset identifier (e.g., 'msmarco', 'nq', 'hotpotqa')
passage	str	Conditional	The passage text for query generation
query	str	Conditional	The query text for answer generation or QC
use_examples	bool	Conditional	Whether to include few-shot examples in prompt

Outputs

Name	Type	Description
prompt	str	Formatted prompt string ready for LLM input

Key Data Structures

Dataset Configuration Dictionaries

# Task descriptions for 25+ datasets
TASK_DICT = {
    'msmarco': 'Given a web search query, retrieve relevant passages that answer the query.',
    'nq': 'Given a question, retrieve Wikipedia passages that answer the question.',
    'hotpotqa': 'Given a multi-hop question, retrieve documents that can help answer the question.',
    # ... 22 more datasets
}

# Query type specifications
QUERY_TYPE_DICT = {
    'msmarco': 'web search query',
    'fever': 'claim',
    'fiqa': 'financial question',
    # ... maps dataset to query type
}

# Passage type specifications
PASSAGE_TYPE_DICT = {
    'msmarco': 'passage',
    'scidocs': 'paper abstract',
    'hotpotqa': 'document',
    # ... maps dataset to passage type
}

Query Generation Parameters

# Query length distribution (weighted random selection)
QUERY_LENGTH_LIST = [
    'less than 5 words',  # 2x probability
    '5 to 10 words',      # 5x probability
    '10 to 15 words',     # 4x probability
    'at least 15 words'   # 1x probability
]

# Query style options
QUERY_STYLE_LIST = [
    'plain and simple',        # 4x
    'common and formal',       # 4x
    'casual and informal',     # 2x
    'professional and complex' # 1x
]

Usage Examples

Generate Query from Passage

import random
from research.Reinforced_IR.data_generation.prompts.get_prompts import get_query_generation_prompt

# Example passage from MS MARCO
passage = """CPAP is a treatment that uses mild air pressure to keep your breathing
airways open. It involves using a CPAP machine that includes a mask or other device
that fits over your nose or your nose and mouth."""

# Generate prompt with examples
prompt = get_query_generation_prompt(
    dataset_name='msmarco',
    passage=passage,
    use_examples=False  # Set to True to include few-shot examples
)

# Send to LLM to generate query
# Expected output: "what is cpap therapy"

Validate Query-Passage Relevance

from research.Reinforced_IR.data_generation.prompts.get_prompts import get_quality_control_prompt

query = "what is cpap therapy"
passage = "CPAP is a treatment that uses mild air pressure..."

# Generate QC prompt
qc_prompt = get_quality_control_prompt(
    dataset_name='msmarco',
    query=query,
    passage=passage
)

# Send to LLM for binary relevance judgment
# Expected output: "1" (relevant) or "0" (not relevant)

Generate Answer for Query

from research.Reinforced_IR.data_generation.prompts.get_prompts import (
    get_additional_info_generation_prompt,
    get_additional_info_generation_long_prompt
)

query = "How does coronavirus differ from seasonal flu?"

# Generate brief answer prompt
prompt = get_additional_info_generation_prompt(
    dataset_name='trec-covid',
    query=query
)

# Or generate longer ~100 word answer
long_prompt = get_additional_info_generation_long_prompt(
    dataset_name='trec-covid',
    query=query
)

Rerank Query-Passage Pairs

from research.Reinforced_IR.data_generation.prompts.get_prompts import get_reranker_prompt

query = "renewable energy sources"
passage = "Solar, wind, and hydroelectric power are renewable energy sources..."

# Generate reranking prompt
rerank_prompt = get_reranker_prompt(
    dataset_name='arxiv',
    query=query,
    passage=passage
)

# Send to LLM for relevance score 0-4
# 0=not relevant, 4=highly relevant

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment